Tm1py: TM1 Git integration

[ghost]

I was going to start building out a GitService to incorporate the new features available in 2.07. Before I get started I wanted to make sure I wasn't duplicating work that was already underway.

Cheers.

[MariusWirtz]

Soungs great!
I wasn't planning to work on this feature in the next weeks. I would be happy for you to lead the way.

[rclapp]

Sounds good, I have started it on my fork. I finally fixed my multiple Git Accounts, so I will work via this branch

https://github.com/rclapp/TM1py/tree/feature_tm1_git

[luiz-gr]

Hi! First I'd like to say that TM1py is very useful as working with postman is very annoying and not practical, so thank you for your hard work!!!

I also wanna know if there's any updates on the git feature, when can we expect it to be released? I'm not demanding anything, just curious because I'm anticipating this!! Also even though my skills are not on the same level as yours, if there's anything I can help you guys with, I can try. I'm a fast learner!
Thank you.

[rclapp]

I have some working code, but I could use some help if you want to contribute.

Ryan Clapp
AWS Business Systems
Sr. Manager

Sent from my mobile device

On Dec 13, 2019 5:43 AM, LuizR-AMM [e-mail] wrote:

Hi! First I'd like to say that TM1py is very useful as working with postman is very annoying and not practical, so thank you for your hard work!!!

I also wanna know if there's any updates on the git feature, when can we expect it to be released? I'm not demanding anything, just curious because I'm anticipating this!! Thank you.

-
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156?email_source=notifications&email_token=AEK7GZQAUMEPZ7JOTIFTDMDQYOGMZA5CNFSM4IDTHUAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2AEYA#issuecomment-565445216, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZTGXWK3ORZMWCD5PFTQYOGMZANCNFSM4IDTHUAA.

[luiz-gr]

I have some working code, but I could use some help if you want to contribute. Ryan Clapp AWS Business Systems Sr. Manager
…
Sent from my mobile device On Dec 13, 2019 5:43 AM, LuizR-AMM [e-mail] wrote: Hi! First I'd like to say that TM1py is very useful as working with postman is very annoying and not practical, so thank you for your hard work!!! I also wanna know if there's any updates on the git feature, when can we expect it to be released? I'm not demanding anything, just curious because I'm anticipating this!! Thank you. - You are receiving this because you commented. Reply to this email directly, view it on GitHub<#156?email_source=notifications&email_token=AEK7GZQAUMEPZ7JOTIFTDMDQYOGMZA5CNFSM4IDTHUAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2AEYA#issuecomment-565445216>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZTGXWK3ORZMWCD5PFTQYOGMZANCNFSM4IDTHUAA.

How can I help?

[rclapp]

@LuizR-AMM can you see this branch?

https://github.com/rclapp/TM1py/tree/feature_tm1_git/

I need to update the branch 1.4 but if you want to start contributing that would be great.

[MaaYuu]

Is there any update about this feature? Is any addition made after last commit that is seen here?
@rclapp

[ghost]

No recent updates. After hitting some bugs on the server I stopped. You're
welcome to pick up from where I left off though.

On Sat, Apr 11, 2020, 9:21 PM MaaYuu [e-mail] wrote:

Is there any update about this feature? Is any addition made after last
commit that is seen here?
@rclapp https://github.com/rclapp

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cubewise-code/tm1py/issues/156#issuecomment-612562141,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJAF6XJ4T2DQBCDURANX63RME6URANCNFSM4IDTHUAA
.

[MariusWirtz]

Hello @rclapp,

We are planning to implement the TM1 built-in git integration over the next weeks at a client.
We do expect to find some issues and plan to report them with IBM.

You mentioned that you already encountered issues.
For the ones that you did not report with IBM, could you give us a quick hint on how to reproduce them?
In case they still exist, we will report them.

Cheers,

Marius

[rclapp]

Absolutely I worked with Hubert on the testing so none of them were officially reported. Most of them have to do with the REST API and C API not playing nicely

GIT integration uses all of the type validation of the REST API. This causes issues when working with objects created by the C-API. For example:

1. Elements in a subset have to be valid member in a dimension unless GIT fails. Any subset that contains }Rollup…. Will cause a failure in GIT
2. Similarly Subsets can not be Null (According to the REST API), how ever if you build a subset and then the elements it contains are deleted from the dimension, the subset is now null. GIT also throws an error here.
3. There was an issue with attribute values but I don’t recall exactly what the issue was.

I built out a lot of the objects and methods in my fork so please use them if you can. If you need me to merge them somewhere let me know.

Thanks

From: Marius Wirtz [e-mail]
Sent: Tuesday, April 21, 2020 8:49 AM
To: cubewise-code/tm1py [e-mail]
Cc: Clapp, Ryan [e-mail]; Mention [e-mail]
Subject: Re: [cubewise-code/tm1py] TM1 Git integration (#156)

Hello @rclapphttps://github.com/rclapp,

We are planning to implement the TM1 built-in git integration over the next weeks at a client.
We do expect to find some issues and plan to report them with IBM.

You mentioned that you already encountered issues.
For the ones that you did not report with IBM, could you give us a quick hint on how to reproduce them?
In case they still exist, we will report them.

Cheers,

Marius

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156#issuecomment-617257066, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZRTNABU6XIC4AWM7PLRNW55RANCNFSM4IDTHUAA.

[MariusWirtz]

Thanks for your answer. We will look into these points.

Regarding the code, I think it's fine in that branch for the moment.
Ideally, we could finalize in the next weeks and merge it into the TM1py master at some point.

[scrumthing]

Just to let you know some issues we encountered so far:

1. The information that a process has security access is not transferred to GIT. The information is present within the rest api but the json is different in GIT.
2. If you use a tm1project file there are some version where you could make the server crash without any log entry. Boom and it is gone! I have to check back with my colleague to see if he could reproduce it properly.
   Besides that I am looking forward to your implementation because i am too lazy to try it myself and your code is always much better. :-)

[MariusWirtz]

Thanks @scrumthing! I hope we can meet your high expectations :-)

[macsir]

Hi, all
I testing Git these days on PA 2.0.8 and it looks promising. At least, it is a big step forward so far.
Regardless that how awesome the IBM documentation is (I couldn't any a complete guide of how to write tm1project. It is just bits and pieces.), is there a way to upload control objects into Git with wildcards? Or it would be nice to have them at the first place when uploading the whole model into Git.

Wildcards can be used in the object name, if the object is not a control object. An exclamation mark can be prepended to the resource reference to indicate the server to include the object, even when the object is a control object.

[rclapp]

Git integration still has a long way to go. Lots of problems with objects created via the old interfaces. The project file continues to be a challenge since it's not documented well.

I built a small plugin for Arc and took a first pass at git for tm1py that you can play with/expand if you'd like; but as you said the documentation and IBM adoption has just not been enough.

Sent from my mobile device

On May 13, 2020 10:18 PM, tm1sir [e-mail] wrote:

Hi, all
I testing Git these days on PA 2.0.8 and it looks promising. At least, it is a big step forward so far.
Regardless that how awesome the IBM documentation is (I couldn't any a complete guide of how to write tm1project. It is just bits and pieces.), is there a way to upload control objects into Git with wildcards? Or it would be nice to have them at the first place when uploading the whole model into Git.

Wildcards can be used in the object name, if the object is not a control object. An exclamation mark can be prepended to the resource reference to indicate the server to include the object, even when the object is a control object.

-
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156#issuecomment-628392245, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZTWM4A3D2KFSYP7B5DRRN5F3ANCNFSM4IDTHUAA.

[scrumthing]

@macsir the short (and unsatisfying) answer is NO. But as almost everybody is complaining about that we will hopefully see a change there in the future.
The project file is as @rclapp said a mostly undocumented mystery. I think you can do a lot with it like different deployments and using it for starting chores and processes from afar without using “oldschool” runti.
I find it helpful to always keep in mind that IBM wants you to start thinking as a software developer for tm1 and you use git to deploy new versions of your application.

[macsir]

Thanks, guys. I always treat myself as a software developer rather than a business consultant, which is IBM's expectation. :)
I found it is a lot easier to use Git branch to deploy what I want than guessing the right format of tm1project file. I have given it up. It is only useful when I want to tell tm1 what needs to be pushed to Git for the first time for initialization where it extracts all object definitions.

[scrumthing]

I totally agree. I haven’t given up either and I still hope with this file I can somehow use one GIT repository to deploy different objects to multiple instances.

[luizg2018]

Did any of you find a way how to deal with attributes? Because they're not part of the git context since they're stored in the .cub files.

[rclapp]

I had played with it. I think we had them written out as a text file and read back in on deployment when we tested.

Thanks,

Ryan Clapp

From: luizg2018 [e-mail]
Sent: Thursday, May 14, 2020 10:22 AM
To: cubewise-code/tm1py [e-mail]
Cc: Clapp, Ryan [e-mail]; Mention [e-mail]
Subject: Re: [cubewise-code/tm1py] TM1 Git integration (#156)

Did any of you find a way how to deal with attributes? Because they're not part of the git context since they're stored in the .cub files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156#issuecomment-628776343, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZQRQXKG6AZLDKTWKZLRRQSDLANCNFSM4IDTHUAA.

[scrumthing]

So far only attributes can be migrated by adding the control cube and control dimension to the not ignore list. The values are treated as data (at least for now) and therefore do not belong in GIT. I agree that this causes a lot of pain.
If you do not already load them from files or another database the only way is to manually export and import them.

[luizg2018]

Thank you @rclapp and @scrumthing for your answers. I'll try this export/import approach.

I have one more question. Did any of you succeed using the TM1 git as a team, having two separate TM1 servers with the same model and both pushing/commiting to the same repository?

When I tried this, I noticed that if dev A made a change in cube A and then pushed to the repository, then dev b made a change to a cube B and pushed to another branch and tried to merge with the master branch , since dev b still doesn't have the changes on cube A made by dev A, the "old version" of cube A on his model will replace the changes by devA. And if he tried doing a git pull before doing the push, his changes made on cube B will be lost since DEVA didn't have them, like there is no merge conflict since there is no proper "commit" in the TM1 git.

So is there any way to do this or did I do something wrong?

Sorry if I couldn't make my words clear enough as english is not my first language. Thank you.

[scrumthing]

You could try to work with the TM1 project file. Set one up for each server, exclude the objects you are working on and exclude the TM1 project file und then try a pull. Haven’t tried that but could work... theoretically...

[macsir]

Thank you @rclapp and @scrumthing for your answers. I'll try this export/import approach.

I have one more question. Did any of you succeed using the TM1 git as a team, having two separate TM1 servers with the same model and both pushing/commiting to the same repository?

When I tried this, I noticed that if dev A made a change in cube A and then pushed to the repository, then dev b made a change to a cube B and pushed to another branch and tried to merge with the master branch , since dev b still doesn't have the changes on cube A made by dev A, the "old version" of cube A on his model will replace the changes by devA. And if he tried doing a git pull before doing the push, his changes made on cube B will be lost since DEVA didn't have them, like there is no merge conflict since there is no proper "commit" in the TM1 git.

So is there any way to do this or did I do something wrong?

Sorry if I couldn't make my words clear enough as english is not my first language. Thank you.

What I am doing is just to checkout what you need to deploy from the central repo into the branch. You can create branch A for just cube A and create branch B for cube B. At anytime, you are able to merge cube A and cube B back to central repo.

[scrumthing]

Did anyone made any major progress with git so far? Started playing around with it after I installed 2.0.9.3 and still encountering issues and problems. Especially with the tm1project file.

[rclapp]

@rkvinoth and I have made some progress on our Arc plugin. It does most operations now. I think we need to push an updated version to git hub.

The newer versions are better for sure. I haven't touched the python library in quite sometime though.

I am really waiting for IBM to deliver SSH support, then I can really put it to the test.

Ryan Clapp
Sr. Manager
AWS FinTech

Sent from my mobile device

On Oct 29, 2020 1:12 AM, Christoph Hein [e-mail] wrote:

Did anyone made any major progress with git so far? Started playing around with it after I installed 2.0.9.3 and still encountering issues and problems. Especially with the tm1project file.

-
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156#issuecomment-718461226, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZWJVRND4YQXU22ZCFLSNEPVPANCNFSM4IDTHUAA.

[scrumthing]

Would love to see the current ARC plugin.

[zsoltmoravcsik]

+1 to the ARC plugin :) I am strugggling to setup a proper hit integration
since weeks

On 2020. Oct 29., Thu at 9:30, Christoph Hein [e-mail]
wrote:

Would love to see the current ARC plugin.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cubewise-code/tm1py/issues/156#issuecomment-718475467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNEK3DDTDNFNNTHU5V6YADSNERYRANCNFSM4IDTHUAA
.

[rclapp]

Feel free to check it out and contribute as you can. I added @rkvinoth ‘s updates too

https://github.com/rclapp/git-for-tm1

From: Zsolt Moravcsik [e-mail]
Sent: Thursday, October 29, 2020 1:33 AM
To: cubewise-code/tm1py [e-mail]
Cc: Clapp, Ryan [e-mail]; Mention [e-mail]
Subject: Re: [cubewise-code/tm1py] TM1 Git integration (#156)

+1 to the ARC plugin :) I am strugggling to setup a proper hit integration
since weeks

On 2020. Oct 29., Thu at 9:30, Christoph Hein <[email protected][e-mail]>
wrote:

Would love to see the current ARC plugin.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cubewise-code/tm1py/issues/156#issuecomment-718475467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNEK3DDTDNFNNTHU5V6YADSNERYRANCNFSM4IDTHUAA
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/156#issuecomment-718477549, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZUZEF62BN36QDQJ3C3SNESCXANCNFSM4IDTHUAA.

[rkvinoth]

Did anyone made any major progress with git so far? Started playing around with it after I installed 2.0.9.3 and still encountering issues and problems. Especially with the tm1project file.

Can you elaborate on the new issues that you are seeing?

[scrumthing]

I will but right now I am not sure if it is a bug or if I am the problem. :-)