Tortoise-orm: Discussion: v1.0.0 target

[grigi]

TODO — Required

• [ ] Timezone support (#103)
• [ ] Ensure generate_schema EXACTLY the same as SHOW CREATE TABLE in MySQL (#237)
• [ ] Good aggregate/filter support (#309, #257)
• [ ] Formal custom select with SQL param placeholder management (#297 #139)

TODO — Nice to have

• [ ] MySQL schema generation to be exact (#237)
• [ ] Change to all-parametrized queries (#81 #151)
• Common framework samples/integration:
  
  
  
  • [ ] aiohttp
  
  
  • [x] sanic
  
  
  • [x] starlette
  
  
  • [x] quart (#119 #121)
  
  
  • [ ] tornado (#99)
  
  
  • [x] fastapi (#188) — starlette + pydantic
  
  
• Documentation: (see https://realpython.com/documenting-python-code/#documenting-your-python-projects)
  
  
  
  • [x] Provide concrete examples of what tortoise improves on (#131)
  
  
  • [ ] Policy re extra DB backends (#31)
  
  
  • [ ] Document app concept
  
  
  • [ ] Document Filters
  
  
  • [x] Document Aggregate Functions (#3)
  
  
• Complete Testing:
  
  
  
  • [x] Test Q (filter, not-Q, bad-Q)
  
  
  • [ ] Test values() & values_list() (with relation, multi-key)
  
  
  • [ ] Test sorting/filtering of M2M relation
  
  

Done

• [x] Add BinaryField type
• [x] Fields Schema generation refactoring (#97, #206)
  
  
  
  • [x] Ability to map ALL field types to custom DB table/field names. (#55)
  
  
  • [x] Schema discovery (#39) (mostly done already, look at #94)
  
  
• [x] ~Add OneToOneField (#236)~
• [x] ~Transaction Isolation (#140, #234)~
  
  
  
  • [x] ~Fix DB pools (#69, #229)~
  
  
• [x] ~MySQL (#37)~
• [x] ~Tortoise.init() refactor (#14 #29)~
• [x] ~JSONField (#20)~
• [x] ~Test framework overhaul (#29 #33)~
• [x] ~Better transaction support (#29 #33)~
• [x] ~Add Python3.7 to CI when PyLint>=2 goes stable (#28)~
• [x] ~Reusable Test framework (#59)~
• [x] ~Un-transactioned isolation (#110 #111)~
• [x] ~Clean up Tortoise exception structures (#24)~
  
  
  
  • [x] ~Replace assert and AssertionError with appropriate Tortoise exception. (#24)~
  
  
  • [x] ~Formalise exception inheritance (#24)~
  
  
• [x] ~Cleanup of "Issues" and "Projects"~
• [x] ~Tortoise Organisation in github~
• [x] ~Simple https://tortoise.github.io/ landing page~
• [x] ~Generic PK field that references real PK field (#36 #122)~
• [x] ~Add description on field (#124)~
• [x] ~Python 3.7 support (#22)~
  
  
  
  • [x] ~Fix pylint/astroid >= 2 (#23)~
  
  
• Testing:
  
  
  
  • [x] ~Test all data fields (#20)~
  
  
  • [x] ~Test .values() and .values_list() (#20)~
  
  
  • [x] ~Test transactions (#29 #33)~
  
  
  • [x] ~Filter tests for data fields (#44)~
  
  
  • [x] ~Test all relation fields (#101)~
  
  
  • [x] ~Filter tests for relation fields (#101)~
  
  
  • [x] ~Test all Model methods (#34)~
  
  
  • [x] ~Test all QuerySet methods (#25)~
  
  
  • [x] ~Test the test classes (#25)~
  
  
  • [x] ~Test the db_url resolver (#25)~
  
  
  • [x] ~Test init (#35)~
  
  
  • [x] ~Test connection resiliency (e.g. stale connections, db restarts, etc...)~
  
  
• Documentation:
  
  
  
  • [x] ~Document init/config (#35)~
  
  
  • [x] ~Document DB_URL (#35)~
  
  
  • [x] ~Marketing material~
  
  
  • [x] ~Contribution guide~
  
  
  • [x] ~Expand documentation re Fields (#25)~
  
  
  • [x] ~Expand documentation re Querysets (#25)~
  
  
  • [x] ~Expand documentation re Filtering~
  
  
  • [x] ~Expand documentation re Exceptions (#25)~
  
  
  • [x] ~Expand documentation re Database backends~
  
  
  • [x] ~Expand documentation re Models~
  
  
  • [x] ~Update README with travis/coveralls badges (#24)~
  
  
  • [x] ~Document usage of Model~
  
  
  • [x] ~Document Tortoise singleton~
  
  
  • [x] ~Document introspection _meta~
  
  
  • [x] ~Cleaned up examples~
  
  

[abondar]

I think goals are good, just need to add fields schema generation refactoring to the list, I really see it as important fix for release

[grigi]

I added that, and expanded the list with more detail, specifically regarding testing.
I will focus on the testing for the foreseeable future.

The list looks quite long...

[grigi]

@Zeliboba5 I updated the initial post with MUST/SHOULD/NICE-TO

You should be able to edit it, so please shuffle things around as you see your priorities?
In the mean time I'll keep on focusing on testing, and anything that makes that better/easier.

[abondar]

I swapped subqueries and .init() refactoring, I think refactoring .init() is something that have the most potential to make some backward incompatible changes, so it would be better to do it before 1.0 release. I think I will look into making something of it in near future. While subqueries is mostly optimisation feature and most users won't even realise that they could use it.

[grigi]

I was deliberately leaving optimisation out for now.
I have learnt the hard way that premature optimisation is a complete waste of time.
So we build a suite of benchmarks (be they contrived or not, at first it doesn't really matter) and then you profile it, and start working on the low-hanging fruit.

I did a talk on it for a project I was working on.
The gist is I spent weeks on optimising it, and only got about 50% faster, the majority was undoing earlier "optimisations". Then I did profiling, and in the two days another 3x speedup.
Surprising issues was things like date parsing, validation and caching stupidly.

But to limit mistakes, we need a very good test suite as well.

I suspect we will probably see big performance issues with the queryset, object instantiation and DB interfacing. Because that is the majority of what this does.
But pointless without knowing if we are moving forward or backwards.

Why am I repeating myself? Here is the talk: https://www.youtube.com/watch?v=GtZgpNlaabI

[grigi]

@Zeliboba5 As per https://github.com/Zeliboba5/tortoise-orm/pull/43#issuecomment-415726943 I feel we should consider moving Tortoise ORM into its own github org before v1.0
The reasoning for this is that Github pages are available for the org or username only, so right now we can do a zeliboba5.github.com, which is silly. I'd rather have a tortoise.github.com

The issue is that there is a username of tortoise already, so we can't use it. (maybe we could ask for it)

What org name would you propose if tortoise is not available?
Or should we use another static site provider? e.g. GitLab pages? tortoise.gitlab.io ?

[abondar]

@grigi Regarding site I can buy tortoise-orm.com, it costs surprisingly low amount.

Organisation name matter is harder one, easy way is tortoise-aio (regular -io postfix doesn't seem to make sense in my eyes) which makes sense a little, especially if we will manage to make more async infrastructure around tortoise. But unpleasant thing about it is aio_libs organisation basically labelled aio as something of their creating and most users will relate such name to them, which might be inconvenient for them.
Also, @Tortoise doesn't provide any form of communication in profile, may be I can contact him through support, even though I don't think it will succeed.

[grigi]

I didn't think of tortoise-io, I think first prize is to try and contact @Tortoise and ask him/her if their profile is defunkt?
I feel we shouldn't use tortoise-aio as it limits the org scope to async stuff. We might actually put something that isn't strictly async?

[grigi]

@Zeliboba5 https://github.com/Tortoise is available for us :-) I invited you

[abondar]

@grigi Yeah, support released it for us.
But I think we should name organisation as tortoise even though I still think we should stick to calling it Tortoise with capital, like https://github.com/django/django does. It just because capital letters looks odd in browser url and github repository name interface

[grigi]

It works as https://github.com/tortoise for me, and you are an owner, so you should be able to change anything :-)

[grigi]

@Zeliboba5 I added https://tortoise.github.io/ to just redirect to the RTD docs for now. I don't have time to build a new site right now, but at least we have a shorter URL now :-)

[grigi]

@abondar In this roadmap there is a " Fields Schema generation refactoring" which I think you wanted (I think).
Could you please elaborate, or better yet, create a ticket with details of what the end-goal should be?

[abondar]

Wrote it down in #97

[grigi]

@abondar I updated the v1.0 list with things I feel is actually necessary for the project to be marked as stable.

I split it into easy/hard for the todos, and added some things to address many of the question issues we are getting recently.

Could you please review and add/change details where you think it is needed?

[abondar]

I think it all looks sane and if we would implement it all - it wouldn't be shameful to call Tortoise stable lib :)
Of "hard" tasks I think our first priority should be pooling, because it still could bring unsolvable issues for users, which is really sad.

[grigi]

Yes. It has been bugging me for a while (evidently)
I have been thinking about the Py3.7 contextvars issue, and I think the context get grabbed a level too high, and we should consider manually managing the context there.

The other very "hard" task is the parametrized queries. I could do create as it was easy to isolate, but don't know how to isolate SELECTS or UPDATES or DELETES as they all get controlled by the Queryset. Can you please have a look at it.
A simple scenario it would enable is a BLOB field, or SQL injection attacks.

Would really love to finally hit the v1.0 :grin:

[csrftoken]

Hello，
When will 1.0 be released? Looking forward to it

[grigi]

Thanks for looking forward to it :-)
We can't predict when it will be done, just getting closer in fits and starts.
Would be great to help us along, even if it is just to get other people to look at it.

[csrftoken]

I really want to help you, but I still feel there are some deficiencies in my ability. I will try my best。
Thanks!