Elasticsearch-dsl-py: Feature roadmap

[3lnc]

Hi. Is there any roadmap/plan for work in dsl/es-py libraries? I may help with smth, but so far I have no idea if there's anything to take a look into.

Just from the top of my head:

1. I remember idea to refactor connections to avoid connections.connections.create_connection monster
2. PEP8 checks over overall, mb adding linter to build
3. Object/Nested interface is kinda strange + may be worth documenting better
4. Make some attrs more explicit/unified - required/multi/dynamic
5. Any features from ES6 not yet implemented?

Is any if those viable?

[HonzaKral]

Thanks so much for the issue, this is much needed and I appreciate your help in this! I highlighted some needed changes in https://github.com/elastic/elasticsearch-dsl-py/issues/771#issuecomment-349676378 for 6.0 changes and general breaking changes I would like to introduce.

To answer your points:

1. that would be awesome if we could come up with better API/naming. This one is cumbersome for sure!
2. I have to say I am not super interested in pep-8 fixes. It is nice to have but not a priority by any means.
3. +1, see my comment referenced earlier
4. not sure what you mean here, any suggestions, examples?
5. no feature is missing but perhaps we should re-examine how we deal with indices/types (again see some details in the comment above).

[3lnc]

1. I propose to remove elasticsearch_dsl.connections.Connections, pull all methods up into module scope. Same for _kwargs && _conns (leaving them "private"). Interface may look like

from elasticsearch_dsl import connections

connections.create/add/remove/whatever_connection(<params>)

Basically, we will use connections module as ready-made singleton. We don't have any [re]write protection for Connections._conns anyway.

If we'll came up with some idea, this definitely should be merged before cut to 6.x

1. I propose to pull such params to arguments in __init__ as much as possible. Maybe I'm missing some metaclass stuff that can be broken, but having proper introspection for

class Field(DslBase):
    def __init__(self, multi=False, required=False, *args, **kwargs):
        self._multi = multi
        self._required = required
        super(Field, self).__init__(*args, **kwargs)

is way better than current interface with

class Field(DslBase):
    def __init__(self, *args, **kwargs):
        self._multi = kwargs.pop('multi', False)
        self._required = kwargs.pop('required', False)
        super(Field, self).__init__(*args, **kwargs)

1. May way out of scope for actually 6.x, but I had some fuzzy idea of some Index object as meta instead of plain text. Think

class Doc(DocType):
    f = Field()
    class Meta:
        index = Index(name='blah', coerce=False, ...)

To be able to describe index-wide params and/or to get some proxy to elasticsearch.client.IndicesClient via Doc.index
Alternatively, if we aim to es7 at some point in future, DocType should be refactored to Index or smth, to support index-level params in Meta

[HonzaKral]

I implemented connections in the connections branch via 0729a42e118bf10b0acf34289a223bf340e87f7c, thanks for the idea, I didn't realize how simple it would be to create a shortcut to the singleton we already have while keeping things backwards compatible. I will add some tests and merge it later.

[HonzaKral]

Thank you again for the issue @3lnc, I will close it now and open an issue for every individual step/feature we want to do (with a discuss label), which would now be primarily to figure out Meta.index since I really like your idea of allowing more structure there (and with deprecation of types in 6.0 it makes even more sense, maybe it can replace the Index class altogether?). Feel free to do the same for any ideas you have.

[HonzaKral]

see #800 for the Meta.index discussion, any and all feedback is welcome!