Ts-toolbelt: Support dotted path notation

[nandorojo]

🍩 Feature Request

Is your feature request related to a problem?

TypeScript 4.1 has template literals, and they're super useful. However, the only solutions out there are quite hacky to use, and result in some bugs.

I want to be able to get a nested object path, as shown in this repo.

Describe the solution you'd like

import { Object } from 'ts-toolbelt'

type I = { hi: { hey: true } }

type Nested = Object.NestedPath<I, 'hi.hey'>

I want to know how do I get the nested object path using dot notation of an object. I'd then like to get the value that matches that object with the path.

I'd like to achieve this:

This is the idea:

type Path<T, Key extends keyof T = keyof T> =
Key extends string
? T[Key] extends Record<string, any>
  ? | `${Key}.${Path<T[Key], Exclude<keyof T[Key], keyof Array<any>>> & string}`
    | `${Key}.${Exclude<keyof T[Key], keyof Array<any>> & string}`
    | Key
  : never
: never;

type PathValue<T, P extends Path<T>> =
  P extends `${infer Key}.${infer Rest}`
  ? Key extends keyof T
    ? Rest extends Path<T[Key]>
      ? PathValue<T[Key], Rest>
      : never
    : never
  : P extends keyof T
    ? T[P]
    : never;


declare function get<T, P extends Path<T>>(obj: T, path: P): PathValue<T, P>;

However, this code block above seems to be buggy. I occasionally get infinite loop errors from typescript if I use it (but not always), so someone who has a better understanding of TS than I do should probably make it.

Describe alternatives you've considered

I tried the code sample above. However, it is not very robust. Someone put it on twitter, but I think it would make more sense for it to live in TS toolbelt (which is always so helpful.)

Teachability, Documentation, Adoption, Migration Strategy

This is a great article describing the template literals. Surprisingly, if you google "TypeScript template literals," it's hard to find content, even though it's such a cool addition. Do you think you might be able to help @millsp? Thank you so much!

[millsp]

¡Hola Nando! ¿Como mola?

We already have something similar here https://millsp.github.io/ts-toolbelt/modules/_object_paths_.html

I can start from there and do this for you in the next week, with a better implementation that won't crash on circular references and that is certainly easier to read and understand/maintain :)

[nandorojo]

Todo bien!

Thank you, great to hear. A key part of the new syntax is that it enforces strict paths for nested strings. I use object paths, but I don't think it quite does the same for nested for notation of only certain paths.

Thanks again!

[nandorojo]

PS I think your link is broken

[millsp]

Fixed

[millsp]

but I don't think it quite does the same for nested for notation of only certain paths

what do you mean?

[nandorojo]

Notice how this allows post.type, but it should only allow post.url, or car.type. Code here.

The correct behavior would be like this:

Code here.

[millsp]

Yep, it works correctly, though. The problem here is that TypeScript replaced ['post', ''] as a type [string, ...] because it widens primitive types in objects - which can definitely be confusing and impractical in your scenario. Reason why we'd rather use the "dot" notation with strings, since string parameters don't undergo such widening.

The implementation I'll work on will have a common limitation with the implementation you linked above: it cannot handle circular references. When this case happens, it won't crash, instead, it will allow any path and not provide auto-completion. Maybe you see a different way of handling this scenario - let me know.

[millsp]

Never mind, I was able to solve this limitation by preventing excess depth. It actually improved the previous implementation - so we can now enjoy circular references to a certain extent.

[millsp]

I limited the output to 10, that's when ts starts complaining that it's too deep (usually 11). I'm going to think over this in the next week to provide you with the full solution.

[nandorojo]

Wow very cool! Is there any shot you could share your temporary solution here to try in the meantime?

Thank you again.

[millsp]

Giving it to you "as is", let me know if we can improve it. I embeds a few safe-guards already. Try to crash it please :1st_place_medal:

import {A, I, L, M} from 'ts-toolbelt'

type _Paths<O, Paths extends L.List<A.Key> = [], Limit extends I.Iteration = I.IterationOf<'0'>> =
  10 extends I.Pos<Limit>
  ? Paths
  : O extends M.BuiltInObject
    ? Paths
    : O extends object
      ? Paths | {
          [K in keyof O]: _Paths<O[K], L.Append<Paths, K>, I.Next<Limit>>
        }[keyof O]
      : Paths

export type Paths<O extends object> =
    _Paths<O>

type Joinable = string | number | boolean | bigint

type Join<
  L extends L.List<Joinable>,
  D extends Joinable,
  J extends Joinable = '',
  I extends I.Iteration = I.IterationOf<'0'>> = {
    0: Join<L, D, `${J}${D}${L[I.Pos<I>]}`, I.Next<I>>
    1: J extends `.${infer J}` ? J : never
}[A.Extends<I.Pos<I>, L.Length<L>>]

type StringPaths<O extends object> =
  Paths<O> extends infer P
  ? P extends unknown
    ? Join<A.Cast<P, L.List<string | number>>, '.'>
    : never
  : never

declare function get<O extends object, P extends StringPaths<O>>(obj: O, path: P | String): P;

declare const object: O

get(object, '')

type O = {
  h: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  b: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  c: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  d: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  }
}

[millsp]

Heya, I woke up fresher this morning. Here's a shorter, cleaner version:

import {I, M} from 'ts-toolbelt'

type _PathsDot<O, Paths extends string = '', Limit extends I.Iteration = I.IterationOf<'0'>> =
  11 extends I.Pos<Limit> ? Paths :
  O extends M.BuiltInObject ? Paths :
  O extends object ? Paths | {
    [K in keyof O]: _PathsDot<O[K], `${Paths}.${K & string}`, I.Next<Limit>>
  }[keyof O]
  : Paths

export type PathsDot<O extends object> =
  _PathsDot<O> extends `.${infer P}` | infer _
  ? P
  : ''

declare function get<O extends object, P extends PathsDot<O>>(obj: O, path: P | String): P;

declare const object: O

get(object, '')

type O = {
  h: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  b: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  c: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  },
  d: {
    b: {
      c: {
        d: {
          e: {
            f: O
          }
        }
      }
    }
  }
}

Notice that I added String into the get definition because I felt like we should be able to pass unknown strings too.

[nandorojo]

Wow, this looks great, I'll test it and report back. Thank you so much!

[millsp]

Thanks, I'll include this in the next release - around the 15th of November. I will also create PathDot so that we can get the type at the end of that path.

[nandorojo]

@millsp This seems to be working great! The only thing is my linter is giving me errors when using String. Should I be importing that from ts-toolbelt?

Let me know if I should just ignore that.

Also, it doesn't seem to support objects nested in arrays at the time.

Not the end of the world, but figured I'd mention it in case you'd know of a solution.

I will also create PathDot so that we can get the type at the end of that path.

Awesome! I assume this means the illustrative get function would return the actual type, rather than P?

Really great work on this overall. ts-toolbelt is always making my life so much easier. Thanks so much.

[millsp]

Also, it doesn't seem to support objects nested in arrays at the time.

I saw that earlier, I updated the snippet :) It should work now.

Awesome! I assume this means the illustrative get function would return the actual type, rather than P

Yes, I just left it like that for now. You can probably use the other implementation for now, though I'm not sure it's 100% type-safe.

Let me know if I should just ignore that.

Yep. Because if I used | string it would swallow all the paths into string. since String is an object, this swallowing behavior does not happen.

Really great work on this overall. ts-toolbelt is always making my life so much easier. Thanks so much.

Amazing! Thanks :) Don't hesitate to share this project with others!

[millsp]

I see one case where the paths cannot be resolved is where you have a list of unknown size - because it could be empty (and I cannot generate a path for all possible indexes). The solution could be to take steps of get every time the autocomplete falls short. Not sure if this was very clear, let me know.

[millsp]

Great news. While programming today, I actually noticed that TypeScript DOES accept to have strings like prop.[number].prop. This is really amazing, because we don't have to worry anymore, it works with arrays too:

https://gist.github.com/millsp/1eec03fbe64592c70efa4c80515f741f

[nandorojo]

Would this work if I did hi[0] syntax too?

[millsp]

No, we would need to implement something. I quickly bootstrapped something and since it increases complexity, you would be limited to 6 nested properties instead of 9.

[millsp]

If that's ok, I can do that for you (are you building a lib), or was it purely out of curiosity? @nandorojo

[nandorojo]

A few different places: I'm using it in my actual app, and I'm also working on adding strict types to Drispy, a design system I maintain. I want to safely use fields such as theme.colors.primary throughout my app, including nested fields, arrays, and such. We've been working on this here for theme-ui, which Dripsy relies on.

[nandorojo]

If that's ok, I can do that for you (are you building a lib), or was it purely out of curiosity? @nandorojo

That should still work, I would imagine that 6 nested properties should be sufficient, and we can fallback to any non-nullable field at that point. Thank you again!!

[nandorojo]

@millsp one minor note – the latest gist also adds length to the suggestions:

For the following type:

type UPBInfo = {
  university: string
  organization_name: string
  test: {
    org: string
  }
}

[millsp]

Good point, I updated the gist so that it doesn't pick up array properties or built-in object properties.

type Obj = {
    ['🦜']: Obj[],
    ['🥜']: { ['🍊']: Obj},
    ['🌴']: 40
    ['🌞']: 50
};

Let me know if it works for you, I haven't thoroughly tested this!

[nandorojo]

The gist is working well for me in development. However, when I run yarn tsc, I'm seeing this:

/Users/path/to/node_modules/typescript/lib/tsc.js:86999
                throw e;
                ^

RangeError: Maximum call stack size exceeded
    at getBaseConstraintOfType (/Users/path/to/node_modules/typescript/lib/tsc.js:43716:41)

This is the code I'm using:

import {
    A,
    F,
    B,
    O,
    I,
    L,
    M
} from 'ts-toolbelt';

type OnlyUserKeys<O> =
    O extends L.List
    ? keyof O & number
    : O extends M.BuiltInObject
      ? never
      : keyof O & (string | number)

type PathsDot<O, I extends I.Iteration = I.IterationOf<'0'>> =
    9 extends I.Pos<I> ? never :
    O extends object
    ? {[K in keyof O & OnlyUserKeys<O>]:
        `${K}` | `${K}.${PathsDot<O[K], I.Next<I>>}`
      }[keyof O & OnlyUserKeys<O>]
    : never

// this breaks
export type DottedPaths<O extends object, P extends string = string> = A.Cast<P, PathsDot<O>>

// this works
export type DottedPaths<O extends object, P extends string = string> = keyof P | (string & {})

[nandorojo]

This may be because I'm using it in Formik. Not sure if that adds something to it. I'll see if a simpler usage breaks it too.

[millsp]

Ok, there's a few things you could test:

• Does the previous gist work or does it fail too? If not =>
• Does it work when you replace M.BuiltInObject by Function?
• Is the variable you are passing a very big type?

Maybe a repro or a link to a branch on your repo, then I could take a look if all else fails?

[nandorojo]

Thanks, I'll try these out!

Yes, it's a very large type so I assume that's probably it. I'll try to reproduce it.

[millsp]

No problem, I'd really like this to work! Just curious, is it a large union or a large object type?

[nandorojo]

Large object, no unions.

[nandorojo]

Although there may be a problem with a large union too, I’ll investigate
more closely.

[jessetabak]

The gist is working well for me in development. However, when I run yarn tsc, I'm seeing this:

```shell
/Users/path/to/node_modules/typescript/lib/tsc.js:86999
throw e;
^

RangeError: Maximum call stack size exceeded
at getBaseConstraintOfType (/Users/path/to/node_modules/typescript/lib/tsc.js:43716:41)

Hi,

I've been following this conversion with great interest and I have the exact same problem in my project. Took me a while to create a repro. It only fails when used inside a generic class and an if statement and an assignment, pretty strange.

import { A, B, O, I, L, M } from 'ts-toolbelt';

export type OnlyUserKeys<T> =
T extends L.List
    ? keyof T & number
    : T extends M.BuiltInObject
        ? never
        : keyof T & (string | number)

export type PathsDot<T, U extends I.Iteration = I.IterationOf<'0'>> =
9 extends I.Pos<U>
    ? never
    : T extends object
        ? { [K in keyof T & OnlyUserKeys<T>]
            : `${K}` | `${K}.${PathsDot<T[K], I.Next<U>>}`
        }[keyof T & OnlyUserKeys<T>]
        : never

type DottedPathString<T extends object, P extends string = string> = A.Cast<P, PathsDot<T>>;

interface IField<T extends object> {
    field?: DottedPathString<T>;
}

// FAILS
class Bar1<T extends object> {
    a: IField<T>;
    b = '';

    go() {
        if (this.a.field) {
            this.b = this.a.field;
        }
    }
}

// SUCCEEDS
class Bar2<T extends object> {
    a: IField<T>;
    b = '';

    go() {
        if (this.a.field != null) {
            this.b = this.a.field;
        }
    }
}

// SUCCEEDS
class Bar3<T extends object> {
    a: IField<T>;
    b = '';

    go() {
        if (this.a.field) {
            this.b = 'x';
        }
    }
}

// SUCCEEDS
class Bar4<T extends object> {
    a: IField<T>;
    b = '';

    go() {
        this.b = this.a.field;
    }
}

[millsp]

Thanks for the insight @jessetabak, I'm going to look into this

[sproott]

I changed this implementation to be an array of strings instead of a single string separated by dots.

export type PathsDot<O, Paths extends List = [], Limit extends I.Iteration = I.IterationOf<'0'>> =
  11 extends I.Pos<Limit> ? Paths :
    O extends M.BuiltInObject ? Paths :
      O extends object ? Paths | {
        [K in keyof GetType<O>]: PathsDot<GetType<O>[K], L.Append<Paths, K & string>, I.Next<Limit>>
      }[keyof GetType<O>]
        : Paths

For a generic type, it gives me for example [] | ["b"] | ["b", "id"] | ["id"] | ["name"]. Now what I want is to be able to extract the current type this has been assigned, for example only ["b"]. This doesn't work though. After calling O.Path on it it converts every one of these possible values to the resulting type, yielding something like string | number | Obj | B which is not what I want.

Anyone knows how to do this?

[jessetabak]

I changed this implementation to be an array of strings instead of a single string separated by dots.

export type PathsDot<O, Paths extends List = [], Limit extends I.Iteration = I.IterationOf<'0'>> =
  11 extends I.Pos<Limit> ? Paths :
    O extends M.BuiltInObject ? Paths :
      O extends object ? Paths | {
        [K in keyof GetType<O>]: PathsDot<GetType<O>[K], L.Append<Paths, K & string>, I.Next<Limit>>
      }[keyof GetType<O>]
        : Paths

For a generic type, it gives me for example [] | ["b"] | ["b", "id"] | ["id"] | ["name"]. Now what I want is to be able to extract the current type this has been assigned, for example only ["b"]. This doesn't work though. After calling O.Path on it it converts every one of these possible values to the resulting type, yielding something like string | number | Obj | B which is not what I want.

Anyone knows how to do this?

It sounds like you want what O.Paths\

[sproott]

Actually I'm sorry, didn't realize, it is exactly what O.Paths does. What I'm really asking about is if I pass a specific value like ["b"], I want the type of that to be only ["b"], not all the possible combinations.

The function:

get<TBase extends object, TKeys extends O.Paths<TBase>, TResult extends O.Path<TBase, TKeys>>
(obj: TBase, keys: TKeys): TResult

The only way I manged to call this with the right type inference was to pass the keys as ["b"] as const (a readonly array). Is it possible to tell that this has to be readonly and not have to pass it as const at every call?

[jessetabak]

It's not exactly what you're looking for, but I use the following in my projects, it does not have the issue you encounter, maybe you can adjust to your needs:

function pathOf<
    T extends { [x: string]: any },
    K1 extends keyof T
>(
    root: T,
    key1: K1
): [T, K1];
function pathOf<
    T extends { [x: string]: any },
    K1 extends keyof T,
    K2 extends keyof T[K1]
>(
    root: T,
    key1: K1,
    key2: K2
): [T, K1, K2];
function pathOf<
    T extends { [x: string]: any },
    K1 extends keyof T,
    K2 extends keyof T[K1],
    K3 extends keyof T[K1][K2]
>(
    root: T,
    key1: K1,
    key2: K2,
    key3: K3
): [T, K1, K2, K3];


function pathOf<
    T extends { [x: string]: any },
    K1 extends keyof T,
    K2 extends keyof T[K1],
    K3 extends keyof T[K1][K2]
>(
    root: T,
    key1: K1,
    key2?: K2,
    key3?: K3
) {
    return [root, ...[key1, key2, key3].filter(Boolean)];
}

interface Foo {
    a: {
        b: {
            c: string;
        }
    },
    d: {
        e: {
            f: number;
        }
    }
}

const foo: Foo = null;

const result = pathOf(foo, 'a', 'b', 'c'); // result == [foo, 'a', 'b', 'c']

I tried to make this work without the first root object parameter, but it does not work, it's this code (short version):

function pathOf<
    T extends { [x: string]: any },
    K1 extends keyof T = keyof T,
    K2 extends keyof T[K1] = keyof T[K1],
    K3 extends keyof T[K1][K2] = keyof T[K1][K2]
>(
    key1: K1,
    key2?: K2,
    key3?: K3
) {
    return [key1, key2, key3].filter(Boolean);
}

const result = pathOf<Foo>('a', 'b', 'c'); // expected result: ['a', 'b', 'c'], but does not work

[johnpyp]

Adding onto this, I had a need to get the type of the target in the object, which I added like this:

// From millsp's reply
type _PathsDot<
  O,
  Paths extends string = "",
  Limit extends I.Iteration = I.IterationOf<"0">
> = 11 extends I.Pos<Limit>
  ? Paths
  : O extends M.BuiltInObject
  ? Paths
  : O extends object
  ?
      | Paths
      | {
          [K in keyof O]: _PathsDot<O[K], `${Paths}.${K & string}`, I.Next<Limit>>;
        }[keyof O]
  : Paths;

export type PathsDot<O extends object> = _PathsDot<O> extends `.${infer P}` | infer _ ? P : "";

// Added value resolving type
export type PathsDotValue<O, P> = P extends keyof O
  ? O[P]
  : P extends `${infer K}.${infer R}`
  ? K extends keyof O
    ? O[K] extends object
      ? PathsDotValue<O[K], R>
      : O[K]
    : never
  : never;

Which lets you write a full get with dotpath notation like this:

declare function get<O extends object, P extends PathsDot<O>>(obj: O, path: P | String): PathsDotValue<O, P>;

I'm not too experienced with advanced typescript stuff like this, so it might be a lot easier than what I did but this seems to work with correct type inference and editor autocomplete. I look forward to seeing these kinds of types added first-class to TS-toolbelt too!

[andreialecu]

I'm also really interested in this. I've read the entire thread, and noticed that this was supposed to be released in November.

Is it blocked because of the typescript crash? If so, I think that should be reported on the typescript issue tracker. Seems like a bug in typescript.

[nandorojo]

I've faced many issues trying to use TS 4.1 and template string literals in my React Native app. If anyone figures out a way for this to work smoothly in an app, please let me know.

[millsp]

Hi all, sorry for leaving this this pending. So if I understood well, there seems to be a problem with the generics as reported by @jessetabak, and @nandorojo is having performance issues on large objects?

I'm willing to fix all the pending problems. If I'm missing something, please add up :)

[andreialecu]

The reason I was looking for this was for strong typing mongodb queries that use dot notation.

With mongodb, when an array is involved, you can omit the array index:
https://docs.mongodb.com/manual/tutorial/query-array-of-documents/#specify-a-query-condition-on-a-field-embedded-in-an-array-of-documents

As per the link above, instock is an array, but both instock.qty and instock.0.qty are valid keys for the query.

Would be great if this feature supported this use case somehow, probably via a special type helper.

[nandorojo]

I've been getting many slow builds in my Next.js app, and after weeks of trying to figure it out, I have a hunch it might be from a memory leak or infinite loop using TypeScript 4.1, notably from string literals.

I have tried many hacky solutions of my own for getting dotted strings to work, so I'm not sure what is causing it, but just something to look out for. I'm going to try to purge all my string literals attempts from my app and see if it changes my build times.

I'm happy to be a tester of the solution you come up with when it's ready. Thanks again for all your help @millsp!

[millsp]

Thanks @nandorojo, have you found your implementations to be efficient and bug free?

If yes, would you be able to share it here?

If not, do you have a repro that I could take a look at?

[millsp]

@andreialecu, I think that the easiest way to do this is to leave the implementation as it is today but write a type that would create a union in such a way that we can either use the one or the other. So it's not directly going to be addressed in this issue.