Designing a robust budget log format

I'm intending to write myself some kind of budgeting software, just for my needs explicitly. I intend to design it "from the format up", as recommended by Linus Torvalds: https://softwareengineering.stackexchange.com/questions/163185/torvalds-quote-about-good-programmer So essentially what I'm trying to do is just "design a robust accounting log format that's capable of my needs [and possibly extensible to future, unforseen needs]", and then from there I just need to write a program to "interface" to it. Basically what I envision from the "user side" of the program is a series of transactions. The transactions can be to or from accounts (possibly between them), and the accounts can have sub-accounts (think domain-style). So for instance, a transaction log might look like <period start="2017-12-25T00:00:00-06:00" end=""> <transaction unit="USD" date="2018-12-01T00:00:00-06:00"> <id type="paycheck.megacorp">2018-01-01</id> <from amount="528.04">WORK.MEGACORP</from> <to amount="142.57">TAX</to> <to amount="35">rfcu.checking.gas</to> <to amount="25">rfcu.checking.groceries</to> <to amount="20">rfcu.checking.propane</to> <to amount="20">rfcu.checking.pocket</to> <to amount="285.47">rfcu.savings</to> </transaction> <transaction unit="USD" date="2018-01-01T08:47:23-06:00"> <id type="receipt.walmart">800700156503</id> <from amount="18.49">rfcu.checking.groceries</from> <from amount="20">rfcu.checking.pocket</from> <to amount="13.42">GROCERIES.FOOD</to> <to amount="5.07">GROCERIES.SCHOOL</to> <to amount="20">wallet.pocket</to> </transaction> <transaction permissive="conversion" date="2018-01-06T22:10:04-06:00"> <id type="ereceipt.coinbase">8YVJD5R9</id> <from amount="200" unit="USD">usaa.savings</from> <to amount="0.0148" unit="XBT">coinbase.gdax</to> </transaction> <transaction unit="USD" date="2018-01-07T08:45:32-06:00"> <comment>Bailout: oops I failed at budgeting</comment> <from amount="252">rfcu.savings</from> <to amount="252">rfcu.checking.rent</to> </transaction> <transaction unit="USD" date="2018-01-10T11:20:52-06:00"> <from amount="252">rfcu.checking.rent</to> <to amount="252">RENT.HOME2</to> </transaction> </period> Now obviously these "sub-accounts" don't refer to "real" accounts which are delimited by the bank in any way; just to budgeting constructs. I'm really hoping if this system would lend itself well to >graphing expenditures, by category would be great. For example a pie chart whose size is the income for that period, and the slices will have say GROCERIES RENT etc >potentially defining "budgets" (<period type="BUDGET"> perhaps for the "template" whose bottom line could be compared with the actual transactions after the period is over? Perhaps even link each real transaction to a specific budget item for accountability?) >someday: just have an app where you can snap picture of receipts as they're printed and it parses budget items for you >"link" accounts together: for example, noting that rfcu.checking.pocket is "the same" account as wallet.pocket, so count transactions between these as essentially trivial and never include them in any graphs Pic related is the handwritten first attempt at logging my expenditures I've fuarrrked it up already somehow but next fortnight will be cleaner hopefully But I would love to have this digitized; if it's a format I've made, especially. It would be cool to be able to log archives long-term and all

(OP here) So what I'm wondering basically is Am I on the right track with this? Would JSON be better? I'm looking for any ideas or feedback on this logging format; obviously it's not complete and needs to include some kind of "initial balance" data

>>6167 >>6166 Have you tried ledger/hledger? As far as I remember it's plaintext, which would let you use gnuplot to plot your expenditures. You would then only have to write the app to ocr your receipt and generate a ledger-compatible output. As far as your logging format is concerned, I don't like it. XML is very complicated in my opinion and if you're going to use your software for years you'll end up with enormous files. I'd advise going with csv instead, it's easier to read, easier to write and it would let you easily replace your plaintext file with an sql database if you ever need it.

>>6166 Why not just use csv? A sub account could just be another column.

>>6168 I'd second csv since it seems to work better for something like this. But check out gnucash which seems to do something like this already and also to my knowledge uses XML as well. It's all open source and has decent documentation.

Log format isn't something you should think about that much. I suggest CSV or TSV (tab-separated values). It's extensible (you can add or remove columns as needed) and easy to write programs to interface with. What Linus means when he says that programmers should worry about data structures and their relationships is not that you should use complex data structures, but that you should design your program around the data you're dealing with, and not around some abstract mental model of the world. Mike Acton says it best, I suggest you watch this talk of his: https://www.youtube.com/watch?v=rX0ItVEVjHc Also, I suggest that when you start programming something you start out with simple data structures, and only switch to something more optimized or sophisticated when you think it'd make the code cleaner. When in doubt, pick simple arrays and structs (or their equivalent in your language of choice).

>>6166 GNUcash uses XML-files IIRC. And its a pretty solid software, I use it for the accounting of my private and business spending .

>all this accounting stuff How/where did you people learn all this stuff? It makes no sense to me

>>6207 it's only complicated if you have complicated finances (extensive lines of credit, stock, etc.). to most people, including myself, "personal finance" pretty much comes down to keeping track of what I spend and what I earn.

>>6166 JSON is much better as a text-based structured data format than xml. However, if you're going to accumulate a large number of values to store, you'll probably want to look into some sql database format.

>>6207 My parents subscribe to Dave Ramsey, so my ideas are basically just a "software" solution to his "Envelopes" idea, but I have perhaps made it more complex than it is normally What they'll do usually is just when money comes in, it's like you're pretending that it's "labeled" based on the budget pie chart But then they actually take the money out of the account and physically label it with envelopes So e.g, instead of cringing "oh my bank account is going to suffer" and putting off refueling (this is irrational behavior since fuel efficiency is slightly improved when the tank is fuller, yet even I suffer from this) It makes it a lot easier to just dispense a few bucks out of the Gas Envelope, which was filled on payday with a fixed amount that you may or may not adjust long-term as suitable As for your question specifically, though, I don't think I've received much "formal" training on how to budget/account, but it basically boils down to >1. make a PLAN for what you intend to do with EVERY dollar in the paycheck. e.g., "save" is fine, but put it in the savings account straightaway so you don't spend it on something else Pie chart is usually useful at this stage, if you understand those easily (And it's fine if your paycheck varies; it doesn't matter how you've planned it out; could use percentages, or maybe fixed values and send all "overflow" to something specific like savings: just so long as you PLAN it) >2. RECORD what you actually did. Just record it. Separately from your bank statement, etc. And then you can go through at the end of the pay period and see >what you're doing >how it compares to what you intended to do >why this is? Do you need to change behavior? Or the plan? Really all you have to do to do "accounting" properly is just record every transaction (preferably including cash) and you're 90% of the way there.

>>6226 >if you're going to accumulate a large number of values to store, you'll probably want to look into some sql database format What I was hoping for, actually, is for each, say, period, or perhaps quarter, to be able to be saved as a separate file (capable of standalone use or viewing together), so that I could back up my budget archives to just e.g., Google Drive SQL databases don't tap into the text-based-format juju, too The idea behind recording transactions, not only just physical transactions but also "virtual reallocation of the imaginary envelopes" (like if I swipe the card for gas and realize I just blew the gas budget, I can just retroactively add "as part of that transaction" when logging it, that I made up the gap with pocket money, rather than just the budget now being fubared), was to make the format such that it would ideally be append-only: you would not normally have any occasion to go back and edit records, you'd just through time be accumulating a sequence of log files that could be transported easily

>>6229 >envelopes >have a plan So it's just taking your money and separating it into groups based on their purpose? What I do is take a set percentage of my income, about 40%, and invest it somewhere for compounding gains; the rest of the money I spend on whatever without thinking or feeling guilty about it. I run out of money sometimes, I never know how much I have either. So to improve my awareness I "transfer" money into each envelope and end up with many of them, each for one purpose? >record what you do So accounting is just like a log from an application? Every time I do something that affects my money, I write down the change? If my money is a variable, I'm supposed to trace all operations that wrote to it? I'm having trouble visualizing the schema for such a database. You guys have all these columns, GNUcash too, excel examples too, when I read about accounting they talk about double entry where you record the same thing in two places and it just feels so redundant. To me it's just rows of money deltas, on time X money +1000, on time X+1 money -50 and so on. My dad is an administrator and he's got some kind of book filled with numbers on multiple columns, I cannot make sense of it.

>>6210 Yeah I wanted to learn more advanced stuff too. When I'm done with school I'm gonna set up my own business so I should probably get educated. I've been reading online and the "accounting algorithm" seems pretty strange, I'm supposed to register transactions without having actually paid or received any money. For example, if I provide a service and charge someone $100 I'm supposed to make note of it and make future decisions as if I had $100 in the bank, even if the customer pays me a ywar later. If I have employees, money borrowed... It got pretty complicated.

>>6230 CSV is basically the same as a SQL database in text format. I strongly suggest you don't pick anything more complex unless you have a very good reason for it, which I don't think you do.

>>6210 Get a good accounting book. Its an important skill to learn and while learning it can be really dry and boring at first, its very useful and not really complicated. Also many people are (forced) to learn it and are often looking for tutors which can be a nice source of income

>>6249 meant to quote >>6207

>>6249 Not him, but do you have recs for a good accounting book? Should I just pickup whatever people use for undergrad? I have a strong background in math so I imagine I could pick up the basic skills rather quickly if I just learn the jargon.

>>6171 >>6173 >csv compared to json or xml, it isn't very extensible, since adding a new column requires either modifying all of your old files or building in some kind of versioning number. if OP is going the table/relational DB route, i'd recommend some kind of SQL database - probably sqlite. >gnucash second this. it's pretty intuitive software, and i'd imagine it would be easy to modify the underlying records with scripts for e.g. email scrapers that add an entry every time you get a new bill from the power company. i think you can choose different backend DB options (iirc, sqlite or xml). i forget if it's a runtime option or if you have to compile one version or the other. >>6167 if it were me, i'd use json over csv. that way you can have all of your little logging scripts writing to the same file regardless of the type of record you're adding.

>>6255 One can also use line-delimited JSON: https://en.m.wikipedia.org/wiki/JSON_streaming It's the best log format I know of: both human readable and very easily parsed by machine. If accounting really is some kind of financial logging then it's a great fit.

>>6255 >since adding a new column requires either modifying all of your old files or building in some kind of versioning number. It really doesn't, you just have to name your columns in the first line and rotate the log if you ever want to add or remove a column, then it works just like a SQL database. Sure, you'll have to deal with different versions when handling them, but that's also true for any other format.

>>6255 I'd second using SQLite, this is a pretty clear usecase for it, and it would scale up pretty well. >>6230 If you really wanted CSV or XML files, you could add an option to export and import from those formats. That way, you could use sqlite for the data you're working with, and another format for archival.