Mastodon: Debian package is necessary.

[matyapiro31]

mastodon has no Debian package.

---

when debian package for mastodon is created, we're able to shrink Dockerfile simply and absorb the difference among version and distribution.

[aral]

I was just about to open an issue for this also. Having a Debian package would make it exponentially easier to set up your own instance and would result in a considerable rise in the instance count.

Imagine:

How to set up Mastodon on Ubuntu (CloudScale.ch, Digital Ocean, etc.)

1. sudo apt-get update
2. sudo apt-get install mastodon

[matyapiro31]

I am expert (but just a few ) of debian package, and maybe it is easy to create for mastodon once, but a package you must maintain takes much time.Can be changed as the way to build be changing.

[ProgVal]

maybe it is easy to create for mastodon once

I am not sure about this. Mastodon has 927 nodejs dependencies and 201 ruby dependencies.

[Gargron]

Mastodon has 927 nodejs dependencies and 168 ruby dependencies

This is not unusual for a project of this scale. This doesn't stop Gitlab from being a debian/ubuntu package.

Generally for this thread: https://github.com/chef/omnibus

[matyapiro31]

I think it may not contain so many packages in node.js and ruby to assign on the package setting directly.
Those package manager also need editing to specify dependencies.

[guillaume-uH57J9]

I am not sure about this. Mastodon has 927 nodejs dependencies and 201 ruby dependencies.

That's good point. I would think twice about installing a package if apt-get asks me "Installing 1128 packages for dependencies. Do you want to continue ? "

Rationalizing nodjs/ruby dependencies in Mastodon (ie reducing their #), or making some of them optional, would go a long way toward packaging.

[aral]

@guillaume-uH57J9 Use case: I want to install Mastodon on my server.

For this use case, I don’t really care what the details of the installation are: Could be 1 dependency, could be 1 million dependencies – the beauty of having a package installer is that I don’t have to care; it’s a black box. My concern is: can I write a single command, hit OK, and boom – a little while later – I have Mastodon installed. Yay! (Much rejoicing, etc.) :)

That said, of course, by all means, please reduce dependencies, etc., if it streamlines the overall process but it’s not a prerequisite for this or the bottleneck that this issue aims to solve. (The bottleneck to new instance adoption is installation complexity and time.) The number of dependencies isn’t really a factor given the tremendous value a package will provide in terms of reducing installation time from hours to minutes – orders of magnitude – and upgrades to a single command.

[matyapiro31]

Most important role of packaging is to separate the environment for running from the env for building.
Can reduce npm package when running isnt what i know; maybe ruby package is unnecessary then.

[aral]

@matyapiro31 Again, as far as my use case for this goes (and I’m happy to open a separate issue if you feel I’m hijacking your use case) :) the most important reason for this is to reduce installation time on the most popular hosting platforms (Ubuntu-based) from hours to minutes (an order of magnitude change that will impact instance adoption rates substantially) and maintenance to a single command. If there are any advantages for developers, that’s an additional bonus but my concern here is in encouraging instance adoption.

[matyapiro31]

For example, http-server is convenient package for testing, but not necessary when run:

$ npm install -g http-server

Mastodon doesn't use http-server , but there are package installed for similar reason.

[matyapiro31]

@Gargron could you list up what packages are used now?

[Gargron]

I do not want to list or justify Mastodon's runtime dependencies. We do not try to make the project bigger on purpose, and everyone who is contributing to the codebase sees that we need the dependencies that we are using. What you are describing ("convenient package for testing, but not necessary when run") is already covered by environments: development, test, production. Dependencies are already sorted accordingly, and production dependencies do not include anything which is not used in production.

You also shouldn't worry about them, I think. What all of these "packaging" tools do, they manage system packages (think postgres, redis, node, protobuf). Ruby and Node dependencies still get installed the usual way, with no difference to manual installation. For a list of system dependencies, look at the production guide: https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Production-guide.md#general-dependencies

[ProgVal]

everyone who is contributing to the codebase sees that we need the dependencies that we are using

I think some of them could be avoided.
I am not very good at Javascript, but I think the dependency on classnames could be removed, for example.

It was added in 860ffc05602b148769b87d0cda39985feb9a8486, apparently just to replace button ${this.props.secondary ? 'button-secondary' : ''} ${this.props.block ? 'button--block' : ''} by:

classNames('button', this.props.className, {
      'button-secondary': this.props.secondary,
      'button--block': this.props.block,
    })

I know it is a small dependency, but it should be possible to remove it at very little cost.

[matyapiro31]

maybe this helps us to create a package.

plus:
/Aptfile and /Vagrantfile can be reduced by writing dependencies.

[matyapiro31]

@Gargron Finally, you think package is necessary?

[akuckartz]

:+1:

[guillaume-uH57J9]

I don’t really care what the details of the installation are: Could be 1 dependency, could be 1 million dependencies

Some users might not care, and it won't change much at installation time.

But for package maintainers this is day and night. Packaging an application + 1 dependency, and maintaining them, is a relatively simple/quick task.
Packaging 1 application + 1 billion dependencies (and maintaining them) cannot be accomplished by one person. It'd require a whole team.

[Jehops]

For the FreeBSD package we punted on the node dependencies. All the OS and ruby/rails dependencies are covered with pkg install mastodon, but then the user still has to do, e.g., sudo su - mastodon; yarn install; RAILS_ENV=production rails assets:precompile. Doing the job of npm/yarn is too ambitious (for me) at this point; we would have to bake a similar infrastructure we have for dealing with ruby packages into the FreeBSD ports/package system.

While it's not a perfect package, most of the feedback I have received from Mastodon administrators running FreeBSD has been positive.

[Gargron]

@Jehops I actually don't really understand why bundler and yarn dependencies cannot be bundled in the package. There is literally no chance that running bundle install or yarn install will yield a different result unless the Mastodon lockfiles change (e.g. in a new release). That's the exact purpose of lockfiles - reproducible, deterministic bundling.

[Jehops]

@Gargron, for each rails dependency, there is a one-to-one correspondence with a FreeBSD port/package and each of those FreeBSD ports/packages handles dependencies. Basically, the FreeBSD port/package system is used in place of gem and bundler. That that sort of solution for the yarn dependencies would be nice, but a lot of work. It may be feasible in the short term to make one omnibus mastodon-yarn-dependencies package.

[matyapiro31]

@Gargron mastodon's package usage design is not better to manage package.
We are forced to use newer version of node, ruby, yarn etc than which in distribution packages.
It makes us very complicated to control mastodon package.
There are at least 5 package varieties:
native packages for mastodon,npm packages,
gem packages, native package for gem,
native packages in sub repositories,
and we must check if those packages has bad affect on other packages, this is the reason why very difficult to maintain package.

[matyapiro31]

I now somehow understand how mastodon works, however it's still very difficult to create debian package.

Mastodon depends on many software that is not packaged correctly for debian. Supplying these packages with appropriate update is very difficult.

[ghost]

1) FYI there is a RFP for this on debian's side https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859741

2) "Mastodon depends on many software that is not packaged correctly for debian" If anyone has identified any of these packages, either this thread or the thread I linked to above would be appropriate places to mention them so we can start filing RFPs and getting those packages in debian, too.

3) "these prerequisites change all the time" That's the whole point of debian - to have a stable basis of development, something that we can actually audit and a known 'good state' that we can build from in case someone puts showstopper bugs in.

[ProgVal]

If anyone has identified any of these packages, either this thread or the thread I linked to above would be appropriate places to mention them so we can start filing RFPs and getting those packages in debian, too.

Ruby:

• [ ] source -> ruby-method-source, ruby-source-map ??
• [x] ruby -> ruby
• [x] pkg-config -> ruby-pkg-config
• [x] puma -> puma
• [x] rails -> ruby-rails
• [x] uglifier -> ruby-uglifier
• [ ] hamlit-rails
• [x] pg -> ruby-pg
• [ ] pghero
• [ ] dotenv-rails
• [x] fog-aws -> ruby-fog-aws
• [x] fog-core -> ruby-fog-core
• [x] fog-local -> ruby-fog-local
• [x] fog-openstack -> ruby-fog-openstack
• [ ] paperclip
• [ ] paperclip-av-transcoder
• [ ] active_model_serializers
• [x] addressable -> ruby-addressable
• [ ] bootsnap
• [x] browser -> ruby-browser
• [ ] charlock_holmes
• [ ] iso-639
• [ ] cld3
• [x] devise -> ruby-devise
• [x] devise-two-factor -> ruby-devise-two-factor
• [x] doorkeeper -> ruby-doorkeeper
• [ ] fast_blank
• [ ] goldfinger
• [x] hiredis -> ruby-hiredis
• [x] redis-namespace -> ruby-redis-namespace
• [x] htmlentities -> ruby-htmlentities
• [x] http -> ruby-http
• [ ] http_accept_language
• [ ] httplog
• [ ] idn-ruby
• [x] kaminari -> ruby-kaminari
• [ ] link_header
• [x] mime-types -> ruby-mime-types, ruby-mime-types-data
• [x] nokogiri -> ruby-nokogiri
• [ ] nsa
• [x] oj -> ruby-oj
• [ ] ostatus2
• [x] ox -> ruby-ox,
• [x] pundit -> ruby-pundit
• [x] rabl -> ruby-rabl, ruby-rabl-rails
• [x] rack-attack -> ruby-rack-attack
• [x] rack-cors -> ruby-rack-cors
• [ ] rack-timeout
• [x] rails-i18n -> ruby-rails-i18n
• [ ] rails-settings-cached
• [x] redis -> ruby-redis
• [ ] mario-redis-lock
• [x] rqrcode -> ruby-rqrcode
• [x] ruby-oembed -> ruby-oembed
• [x] sanitize -> ruby-sanitize
• [x] sidekiq -> ruby-sidekiq
• [ ] sidekiq-scheduler
• [ ] sidekiq-unique-jobs
• [ ] sidekiq-bulk
• [x] simple-navigation -> ruby-simple-navigation
• [ ] simple_form
• [x] sprockets-rails -> ruby-sprockets-rails
• [ ] strong_migrations
• [x] twitter-text -> ruby-twitter-text
• [ ] tzinfo-data
• [ ] webpacker
• [ ] webpush
• [ ] json-ld-preloaded
• [ ] rdf-normalize

nodejs:

• [ ] babel-plugin-preval
• [ ] postcss-smart-import
• [ ] webpack-merge
• [ ] axios
• [ ] autoprefixer
• [ ] babel-plugin-transform-react-remove-prop-types
• [ ] substring-trie
• [ ] babel-plugin-syntax-dynamic-import
• [ ] dotenv
• [ ] react-textarea-autosize
• [ ] cross-env
• [x] rimraf -> node-rimraf
• [ ] emoji-mart
• [ ] array-includes
• [ ] react-redux-loading-bar
• [x] mkdirp -> node-mkdirp
• [ ] react-router-scroll-4
• [ ] postcss-loader
• [ ] font-awesome
• [ ] react-immutable-proptypes
• [ ] react-overlays
• [ ] react-redux
• [ ] babel-plugin-transform-runtime
• [ ] prop-types
• [ ] react-motion
• [ ] http-link-header
• [ ] node-sass
• [ ] intl
• [ ] webpack-bundle-analyzer
• [ ] object-fit-images
• [ ] sass-loader
• [ ] offline-plugin
• [ ] babel-plugin-transform-decorators-legacy
• [ ] intl-messageformat
• [ ] postcss-object-fit-images
• [ ] react-hotkeys
• [x] escape-html -> node-escape-html
• [ ] resolve-url-loader
• [ ] babel-preset-env
• [x] npmlog -> node-npmlog
• [ ] webpack
• [ ] babel-plugin-transform-react-inline-elements
• [ ] babel-plugin-lodash
• [ ] babel-plugin-transform-react-jsx-self
• [ ] immutable
• [ ] tiny-queue
• [ ] redux-immutable
• [ ] classnames
• [ ] babel-loader
• [x] is-nan -> node-number-is-nan
• [x] uuid -> libjs-node-uuid, node-node-uuid
• [x] express -> node-express
• [ ] react-dom
• [ ] webpack-manifest-plugin
• [ ] path-complete-extname
• [x] lodash -> node-lodash
• [ ] babel-plugin-react-intl
• [ ] style-loader
• [ ] react-intl
• [ ] react-toggle
• [ ] pg
• [ ] detect-passive-events
• [ ] requestidlecallback
• [x] redis -> node-node-redis
• [ ] babel-plugin-transform-class-properties
• [ ] stringz
• [ ] reselect
• [ ] extract-text-webpack-plugin
• [ ] compression-webpack-plugin
• [ ] redux
• [ ] babel-plugin-transform-object-rest-spread
• [ ] react-notification
• [x] js-yaml -> node-js-yaml
• [ ] css-loader
• [x] glob -> node-glob
• [ ] mark-loader
• [ ] websocket.js
• [x] punycode -> node-punycode
• [ ] redux-thunk
• [ ] intersection-observer
• [ ] react-swipeable-views
• [ ] babel-plugin-transform-es2015-modules-commonjs
• [ ] throng
• [ ] react
• [ ] es6-symbol
• [ ] babel-plugin-transform-react-jsx-source
• [ ] rails-ujs
• [ ] uws
• [ ] react-router-dom
• [ ] marky
• [x] object-assign -> node-object-assign
• [ ] intl-relativeformat
• [ ] react-immutable-pure-component
• [x] babel-core -> node-babel-core (buster)
• [ ] babel-preset-react
• [ ] precss
• [ ] file-loader

[ghost]

Things you can check off

• [x] (at least for (contrib:buster), sid) babel-core: https://packages.debian.org/hu/sid/node-babel-core
• [x] ruby-dotenv seems to be what could otherwise be called ruby-rails-dotenv ... https://gemnasium.com/gems/dotenv-rails seems to suggest that they are the same.

• [ ] hamlit-rails - status: RFP
• [ ] pghero : status : RFP
• [ ] paperclip : status : RFP
• [ ] ruby-cucumber-expressions (required by paperclip) status : RFP
• [ ] ruby-av (required by paperclip-av-transcoder) status : RFP
• [x] ruby-cocaine(required by ruby-av) status : in debian! only in buster/sid though. https://packages.debian.org/source/buster/ruby-cocaine .
• [ ] ruby-av-transformer status : RFP

Things you can probably add to that list:

• [x] ruby-html2haml (required to build? hamlit-rails) status: in debian.
• [ ] ruby-appraisal (required to build? hamlit-rails) status: in debian, but woefully behind. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882278
• [x] ruby-rspec (required for ruby-appraisal) status in debian: in stable
• [x] ruby-shoulda (required by paperclip) status in debian: in stable
• [x] ruby-mocha (required by paperclip) status in debian: in stable
• [x] ruby-aws-sdk (required by paperclip) status in debian: in stable
• [x] ruby-bourne (required by paperclip) status in debian: in stable
• [x] ruby-cucumber-rails (required by paperclip) status in debian: in stable
• [x] ruby-coveralls (required by ruby-cucumber-expressions) status in debian: in stable
• [x] pry ruby-coderay ruby-method-source (required by ruby-cucumber-expressions) status in debian: in stable
• [x] launchy (required by paperclip) status in debian: in stable
• [x] ruby-fakeweb (required by paperclip) status in debian: in stable
• [x] ruby-generator-spec (required by paperclip) status in debian: in stable
• [x] ruby-simplecov (required by ruby-climate-control ) status in debian: in stable
• [x] ruby-climate-control (required by ruby-cocaine) status : in debian stable