Redash: New developer guide

[arikfr]

Our current developer guide consists of instructions to use a Vagrant box, that seems to be broken by now :-( (well it was created 2 years ago)

I want to create new guide that documents:

• What are the dependencies of Redash
• How to set it up on OS/X or Linux
• New Vagrant box and/or Docker images

Vagrant has the advantage of being a full VM, while Docker has the advantage of being more lightweight and inline with our future production deployments.

---

Topics to document:

• Setting up the dependencies
• How to run the processes
• How to run the frontend build pipeline

[arikfr]

Some draft:

to run Redash you need:

1. Python 2.7
2. Redis
3. PostgreSQL
4. Node.js
   You can install everything w/ brew.

Then create a virtualenv, and install requirements.txt and requirements_dev.txt.

To create the DB:

1. createdb redash
2. ./bin/run python manage.py database create_tables
3. ./bin/run python manage.py users create --admin --password admin "Admin" "admin"

1. All the frontend code is in client.
2. You run npm install from client (will probably move to the top level once I'm done).
3. I run both the flask app (manage.py runserver as before) but also the webpack dev server: npm run start. webpack's devserver builds everything, runs lint and some more. It also watches for changes and reloads the page.
4. The code is structured ok, but you might find some conflicting naming convention here and there. I'm planning to review it all before merging into master.

we also use es6 now and angular 1.5.

[spasovski]

Here's a gist of the instructions I had:

https://gist.github.com/spasovski/3d70f7ddb17457df1c51ad308aa8b921

[rockwotj]

Please make note of node version needing to be at least v4 for things to work and you should connect via localhost:8080

@spasovski that does work now, bower is no longer used and webpack is used.

[rockwotj]

Had to export my variables from .env ./manage.py show_settings was super useful to debug this. I would rather have a config file that was read in than env variables FWIW

Might be useful to specify ./bin/run ./manage.py for those commands

But still getting issues with the database:

  File "/Users/rockwood/Workspaces/redash/redash/permissions.py", line 47, in decorated
    return fn(*args, **kwargs)
  File "/Users/rockwood/Workspaces/redash/redash/handlers/data_sources.py", line 22, in get
    data_source = models.DataSource.get_by_id_and_org(data_source_id, self.current_org)
  File "/Users/rockwood/Workspaces/redash/redash/models.py", line 177, in get_by_id_and_org
    return db.session.query(cls).filter(cls.id == object_id, cls.org == org).one()
  File "/Users/rockwood/Workspaces/redash/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2762, in one
    raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()

It seems I can't save any data sources.

I'm getting a 200 POSTing a data_source, but it's not being persisted:

curl 'http://localhost:8080/api/data_sources' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, */*' -d '{"options":{"dbname":"rockwood"},"type":"pg","name":"foo"}' 

[jeffwidman]

I would rather have a config file that was read in than env variables FWIW

I'm not sure of exact context you're referring to, but in Flask it's not hard to support both. I have an app where it has dev-level defaults hardcoded, which can be overridden by a config file, which in turn is overridden by env vars (highest priority). I can dig up the sample code if needed

[jeffwidman]

As a backend/python guy, the stuff I am least familiar with is getting the webpack stuff going... the db stuff, docker, python venv etc is pretty straightforward and well documented.

The app-specific stuff of course has to be documented, like populating the DB etc.

I am unlikely to touch front-end code much, but I do need to run the app to verify that backend changes I make propagate correctly to the browser. So all I really need on the front-end side is "Here's how to run enough webpack to get the app going with auto-reload and here's links to webpack docs if you want to learn more"...

[rockwotj]

I've had no issues with webpack. I can get the current master branch running, but get a ton of errors when I go to save things to the database. I'm not sure what I've done wrong, as if I save stuff via the CLI it's fine.

For webpack:

• Have at least node v4 installed
• Run npm install from the client directory (where package.json is located)
• Then run npm start
• Point your web browser to localhost:8080 to see the live reloaded webpages

[arikfr]

I've had no issues with webpack. I can get the current master branch running, but get a ton of errors when I go to save things to the database.

@rockwotj all the errors like the one you posted above or there are different ones?

I am unlikely to touch front-end code much, but I do need to run the app to verify that backend changes I make propagate correctly to the browser. So all I really need on the front-end side is "Here's how to run enough webpack to get the app going with auto-reload and here's links to webpack docs if you want to learn more"...

@jeffwidman auto-reload for webpack or Flask?

[arikfr]

I have an app where it has dev-level defaults hardcoded, which can be overridden by a config file, which in turn is overridden by env vars (highest priority). I can dig up the sample code if needed

Yes, please. :)

[rockwotj]

@arikfr yes they are similar. I got a different one trying to save queries when I added the data source via the CLI. Googling it some something about trying to save an id or int when it expected a model object. I'll try and get the stack trace tonight.

[arikfr]

I think you were hitting an issue that I just pushed a fix for in #1484 (9b4cf4d3ee5a41ccaf1a0bd707fe597c1b4554cf). I actually remember fixing it in the SQLAlchemy branch, but somehow the fix was overriden :-(

[rockwotj]

@arikfr that works thanks! However it seems the results aren't showing and I can't create a dashboard, but I'll look into that.

[stefanseifert]

why do we need a dedicated redash vagrant box and do not use just a default one e.g. ubuntu?

i've prepared such a configuration change in this branch and it seems to work fine. just fire up the vagrant box and start webpack and the python app:
https://github.com/stefanseifert/redash/tree/feature/vagrant-dev/setup/vagrant
the README describes the required steps.

i've tested it only running vagrant on a windows host with oracle virtualbox.

[arikfr]

@stefanseifert the main reason was that the Vagrant setup requires many dependencies -- Node.JS, PostgreSQL, Redis and many pip packages (along with system packages). It takes a long time and prone to failures (network failures, temporary missing stuff, etc). So it felt like creating a dedicated box is a good solution for this.

In retrospective, the box itself became a point of failure :-\ This is why I have more faith in a Docker based workflow (where the image is easier to update). I'm just not sure it's comfortable/familiar enough to others.

[arikfr]

First draft of the guide:
https://redash.io/help-onpremise/dev/guide.html

[arikfr]

Related: #1530 (Docker based workflow)

[uptonking]

@arikfr
To run redash on local machine as the dev guide , exec "npm run start" throws error
```
npm ERR! missing script: start
````
did you miss the script in package.json, or we should webpack script to get it to work .
Thanks

[arikfr]

@uptonking are you sure you're using a recent master branch?

[arikfr]

https://github.com/getredash/redash/blob/master/package.json#L7

[uptonking]

@arikfr thanks, I used the "1.0.0 - RC". After git clone the master, "npm run start" works.
another new problem when exec "bin/run ./manage.py runserver --debugger --reload" to start web server.

ImportError: No module named cli

ImportError Pic

[arikfr]

Did you install updated pip packages?

[uptonking]

@arikfr after using virtualenv and pip install again, all the commands in the new dev guide works fine.
But when get localhost:8080，shows internal server error.
the terminal is ProgrammingError: (psycopg2.ProgrammingError) relation "organizations" does not exist
did i miss the db init script to init postgres?
I have executed all commands as the guide.

[arikfr]

To create the tables you need to run:

bin/run ./manage.py database create_tables

[arikfr]

The developer guide has been updated along with new documentation on how to use Docker for easy setup of your dev environment.

https://redash.io/help-onpremise/dev/guide.html

Thank you for all your feedback!
Your feedback on the new guide & flow is always welcomed.