Chef: apt_package :install w/ options broken in 12.19.33

[runningman84]

Description

Since yesterday it is not possible to run test kitchen on any of our cookbooks. The apt_package seems to be broken. Version 12.18.31 and below are working fine. We first saw this issue with 12.19.33.
I suppose this commit broke the apt-get install arguments:
https://github.com/chef/chef/commit/67c44883fda6097385a7d378930177a9c85643b8

Chef Version

12.19.33

Platform Version

ubuntu 14.04

Replication Case

This cookbook has a test kitchen which works fine with 12.18.31 and fails with 12.19.33
https://github.com/runningman84/chef-monitor

Client Output

         * apt_package[sensu] action install[2017-02-23T09:21:33+00:00] INFO: Processing apt_package[sensu] action install (sensu::_linux line 42)


           ================================================================================
           Error executing action `install` on resource 'apt_package[sensu]'
           ================================================================================

           Mixlib::ShellOut::ShellCommandFailed
           ------------------------------------
           Expected process to exit with [0], but received '100'
           ---- Begin output of ["apt-get", "-q", "-y", "--force-yes", "-o", "Dpkg::Options::=\"--force-confdef\"", "-o", "Dpkg::Options::=\"--force-confnew\"", "install", "sensu=0.26.5-2"] ----
           STDOUT: Reading package lists...
           Building dependency tree...
           Reading state information...
           The following NEW packages will be installed:
             sensu
           0 upgraded, 1 newly installed, 0 to remove and 55 not upgraded.
           Need to get 30.0 MB of archives.
           After this operation, 89.0 MB of additional disk space will be used.
           Get:1 http://repositories.sensuapp.org/apt/ sensu/main sensu amd64 0.26.5-2 [30.0 MB]
           Fetched 30.0 MB in 2s (13.8 MB/s)
           STDERR: dpkg: error: need an action option

           Type dpkg --help for help about installing and deinstalling packages [*];
           Use 'apt' or 'aptitude' for user-friendly package management;
           Type dpkg -Dhelp for a list of dpkg debug flag values;
           Type dpkg --force-help for a list of forcing options;
           Type dpkg-deb --help for help about manipulating *.deb files;

           Options marked [*] produce a lot of output - pipe it through 'less' or 'more' !
           E: Sub-process /usr/bin/dpkg returned an error code (2)
           ---- End output of ["apt-get", "-q", "-y", "--force-yes", "-o", "Dpkg::Options::=\"--force-confdef\"", "-o", "Dpkg::Options::=\"--force-confnew\"", "install", "sensu=0.26.5-2"] ----
           Ran ["apt-get", "-q", "-y", "--force-yes", "-o", "Dpkg::Options::=\"--force-confdef\"", "-o", "Dpkg::Options::=\"--force-confnew\"", "install", "sensu=0.26.5-2"] returned 100

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/sensu/recipes/_linux.rb

            42:   package "sensu" do
            43:     version node["sensu"]["version"]
            44:     options package_options
            45:     notifies :create, "ruby_block[sensu_service_trigger]"
            46:   end
            47: when "rhel", "fedora"

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/sensu/recipes/_linux.rb:42:in `from_file'

           apt_package("sensu") do
             package_name "sensu"
             action [:install]
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             declared_type :package
             cookbook_name "sensu"
             recipe_name "_linux"
             version "0.26.5-2"
             options "--force-yes -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confnew\""
           end

           Platform:
           ---------
           x86_64-linux

[cheeseplus]

I can reproduce this with the chef-sensu cookbook with 'kitchen test default-ubuntu-14`

[cheeseplus]

With regard to the chef-sensu cookbook it seems that the package_options it provides on https://github.com/sensu/sensu-chef/blob/develop/recipes/_linux.rb#L40 are related to the issue as removing them results in a normally converging run.

[rmoriz]

Having this issue as well with the following cookbook code on Debian Jessie:

%w(libpq5 postgresql-common).each do |pkg|
  package pkg do
    options "-t #{node['lsb']['codename']}-pgdg -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'"
    action :upgrade
  end
end

Error:

================================================================================
Error executing action `upgrade` on resource 'apt_package[postgresql-common]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '100'
---- Begin output of ["apt-get", "-q", "-y", "-t", "jessie-pgdg", "-o", "Dpkg::Options::='--force-confdef'", "-o", "Dpkg::Options::='--force-confold'", "install", "postgresql-common=180.pgdg80+1"] ----
STDOUT: Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
 postgresql-client-common
The following packages will be upgraded:
 postgresql-client-common postgresql-common
2 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 299 kB of archives.
After this operation, 75.8 kB disk space will be freed.
Get:1 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main postgresql-client-common all 180.pgdg80+1 [79.0 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main postgresql-common all 180.pgdg80+1 [220 kB]
Preconfiguring packages ...
Fetched 299 kB in 0s (1439 kB/s)
dpkg: error: need an action option

Type dpkg --help for help about installing and deinstalling packages [*];
Use 'apt' or 'aptitude' for user-friendly package management;
Type dpkg -Dhelp for a list of dpkg debug flag values;
Type dpkg --force-help for a list of forcing options;
Type dpkg-deb --help for help about manipulating *.deb files;

Options marked [*] produce a lot of output - pipe it through 'less' or 'more' !
STDERR: Reading changelogs...
E: Sub-process /usr/bin/dpkg returned an error code (2)
---- End output of ["apt-get", "-q", "-y", "-t", "jessie-pgdg", "-o", "Dpkg::Options::='--force-confdef'", "-o", "Dpkg::Options::='--force-confold'", "install", "postgresql-common=180.pgdg80+1"] ----
Ran ["apt-get", "-q", "-y", "-t", "jessie-pgdg", "-o", "Dpkg::Options::='--force-confdef'", "-o", "Dpkg::Options::='--force-confold'", "install", "postgresql-common=180.pgdg80+1"] returned 100

Resource Declaration:
---------------------
# In /tmp/kitchen/cache/cookbooks/corp-postgresql/recipes/_postgresql.rb

39:   package pkg do
40:     options "-t #{node['lsb']['codename']}-pgdg -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'"
41:     action :upgrade
42:   end
43: end

Compiled Resource:
------------------
# Declared in /tmp/kitchen/cache/cookbooks/corp-postgresql/recipes/_postgresql.rb:39:in `block in from_file'

apt_package("postgresql-common") do
 package_name "postgresql-common"
 action [:upgrade]
 retries 0
 retry_delay 2
 default_guard_interpreter :default
 declared_type :package
 cookbook_name "corp-postgresql"
 recipe_name "_postgresql"
 options "-t jessie-pgdg -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'"
end

Platform:
---------
x86_64-linux

[rmoriz]

Well… it works on the shell…

root@ci-corp-app:/home/vagrant# apt-get -q -y -t jessie-pgdg -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' install postgresql-common=180.pgdg80+1
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libpq-dev libsensors4 postgresql-client-9.4 sysstat
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  postgresql-common
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 0 B/220 kB of archives.
After this operation, 547 kB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously unselected package postgresql-common.
(Reading database ... 84915 files and directories currently installed.)
Preparing to unpack .../postgresql-common_180.pgdg80+1_all.deb ...
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (180.pgdg80+1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (215-17+deb8u6) ...
Setting up postgresql-common (180.pgdg80+1) ...

Creating config file /etc/postgresql-common/createcluster.conf with new version

Creating config file /etc/logrotate.d/postgresql-common with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Processing triggers for systemd (215-17+deb8u6) ...

[joshbrand]

here's /var/log/apt/term.log first apt via chef-client, then running the same command on the terminal:

```Start-Date: 2017-02-22 20:19:34
Commandline: apt-get -q -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" install sensu=0.26.3-1
Install: sensu:amd64 (0.26.3-1)
Error: Sub-process /usr/bin/dpkg returned an error code (2)
End-Date: 2017-02-22 20:19:34

Start-Date: 2017-02-22 20:21:35
Commandline: apt-get -q -y --force-yes -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew install sensu=0.26.3-1
Install: sensu:amd64 (0.26.3-1)
End-Date: 2017-02-22 20:21:36
```

Looks like a quote escaping issue, maybe due to https://github.com/chef/chef/blob/master/lib/chef/provider/package.rb#L57 ?

[mivok]

Expanding on Josh's comment, it appears that the quotes are being passed in directly when running the command, so to reproduce this on the command line, you would need to run something like:

vagrant@vagrant-ubuntu-trusty-64:~$ sudo apt-get -q -y --force-yes -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confnew\" install sl

(i.e. backslash escape the quotes so that they are seen by apt-get).

My guess is in the older version of chef-client, the command got passed to the shell, or the quotes were otherwise stripped. In the newer version, the quotes aren't stripped by the shell or otherwise, and so apt-get sees them directly.

Interestingly, the quotes around the dpkg options aren't actually needed, so a workaround for now is just to remove them in the affected cookbooks, if possible.

[mivok]

The commit line Josh referred to about the splitting not dealing with quotes could be fixed by using the shellwords module:

[1] pry(main)> require 'shellwords'
=> false
[2] pry(main)> Shellwords.shellsplit('apt-get blah blah Dpkg::Options::="--force-confnew" blah blah')
=> ["apt-get",
 "blah",
 "blah",
 "Dpkg::Options::=--force-confnew",
 "blah",
 "blah"]
[3] pry(main)> 'apt-get blah blah Dpkg::Options::="--force-confnew" blah blah'.split
=> ["apt-get",
 "blah",
 "blah",
 "Dpkg::Options::=\"--force-confnew\"",
 "blah",
 "blah"]