Docker-airflow: Cannot install mssql with current docker image

[AliUz]

I tried adding the mssql package in apache-airflow (pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mssql]==1.8.2) and got the following error:

  Running setup.py bdist_wheel for pymssql: started
  Running setup.py bdist_wheel for pymssql: finished with status 'error'
  Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-q5utw6aw/pymssql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpvfs0qjhbpip-wheel- --python-tag cp36:
  setup.py: platform.system() => 'Linux'
  setup.py: platform.architecture() => ('64bit', '')
  setup.py: platform.linux_distribution() => ('debian', '8.9', '')
  setup.py: platform.libc_ver() => ('glibc', '2.2.5')
  setup.py: Not using bundled FreeTDS
  setup.py: include_dirs = ['/usr/local/include']
  setup.py: library_dirs = ['/usr/local/lib']
  running bdist_wheel
  running build
  running build_ext
  building '_mssql' extension
  creating build
  creating build/temp.linux-x86_64-3.6
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/usr/local/include/python3.6m -c _mssql.c -o build/temp.linux-x86_64-3.6/_mssql.o -DMSDBLIB
  _mssql.c:266:22: fatal error: sqlfront.h: No such file or directory
   #include "sqlfront.h"
                        ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pymssql

Any way to fix this? I need to be able to use the MsSqlHook in my environment.

[AliUz]

By adding freedts-dev to the buildDeps I was able to fix it. Hope this can help someone else out there :)

[alexanderfournier]

@AliUz could you walk me through exactly what you did here. I'm trying to do the same and struggling mightly

[AliUz]

@alexanderfournier I just added the freedts-dev package to the initial buildDeps list and I was able to install airflow with mssql

[alexanderfournier]

@AliUz Do I have to rebuild the container or is there a way to do that post mortem ha. Sorry this is the first time I've used docker so I'm still green. Have you also tried the new odbc connector for mac os yet (they just came out with an alternative ... I can't seem to get that working either lol)

[joshuasheppard]

@alexanderfournier you will need to rebuild the container.

I am trying this as well and also new to Docker. When I follow the "add freetds-dev to the buildDeps list" referenced above it got through (re)building the container.

However, when I actually hit http://localhost:8080/admin/, every DAG that uses MSSQL fails to load with a message like the following:

Broken DAG: [<path-to-dag>] libsybdb.so.5: cannot open shared object file: No such file or directory

It appears that something is missing in the install. I found other pages referencing needing to symlink the libsybdb.so.5 file but I'm not sure where, or if, that needs to be done.

[alexanderfournier]

Hey @joshuasheppard Heads up but Microsoft released an ODBC that works with mac now so you don't need freetds.
Appreciate the feedback I got to the same point you did after some tweaking... stuck where you are.
damn.

[joshuasheppard]

@alexanderfournier - got past the error just now by adding freetds-bin \ before python3-pip \ in the Dockerfile.

In reference to the Microsoft ODBC [driver], have you found a way to use that in this Dockerfile? I'm able to connect from my Mac just fine (last I checked) but wasn't able to in this Debian container (until a few minutes ago).

[joshuasheppard]

Thoughts on a PR with freetds-dev added to the buildDeps list and freetds-bin to the regular dependencies list? If it's useful, I will make the PR (assuming no one beats me to it).

[rolanddb]

@joshuasheppard I'd be interested in that PR. Need to connect to mssql as well..

[kootenpv]

Could #205 be merged?

[kootenpv]

Actually, we also need:

&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,pymssql]==$AIRFLOW_VERSION \

[vlukiyanov]

I've found that to get this to work with Azure SQL you may also need to recompile FreeTDS from scratch with additional options (we have it with --with-libiconv --with-openssl, but not sure both are needed).