Python 3 Thread

Python 3 general. Share resources, PDFs and books. Also discuss and share projects you have been working on. Beginner Questions are also welcome.

Hi everyone, I am in a bit of a pickle (lack of thereof to be honest). I wrote some DataAccessModel like functionality at work. Basic functionality is quite simple, creating and deleting tables and selecting and deleting tables. Because that code is going to be used by my coworkers, I wrote lot of error handling code, cause you know users. In order to reuse as much code as possible, I hevily used decorators for error handling. As an example: def withDataSet(func): def helper(datasetName, *args, **kwargs): datasets = [d.dataset_id for d in client.list_datasets()] if datasetName not in datasets: raise Exception( """dataset {} does not exist in big query instance""" .format(datasetName)) else: return func(datasetName, *args, **kwargs) return helper @withDataSet def createTableBigQuery(datasetName, table, schema=None): bqTable = bigquery.Table( 'irrelevantString.{}.{}'.format(datasetName, table), schema=schema) return client.create_table(bqTable) That is the simplest case, of what is going on, but my main issue is in the insert functionality. I am trying to parallelize it using concurrent.futures, but seems like these closure based decorators cant be pickled[1]. Am I getting something wrong? Do you have any suggestions on how can I rewrite my decorators so the can be pickled. [1]https://stackoverflow.com/questions/45673841/python-attributeerror-cant-pickle-local-object-writebuf-locals-write

>>16054 The error and stack trace are conspicuously missing.

>>16055 Sorry about that. Thanks for pointing it out. Does this one suffice? I do not have the stack trace for the above example, but given it is the exact same than the SO link above, and the overall structure of all the issues is the same, it should provide enough info to point a solution. concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/home/joaquin/anaconda3/lib/python3.7/multiprocessing/queues.py", line 236, in _feed obj = _ForkingPickler.dumps(obj) File "/home/joaquin/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) AttributeError: Can't pickle local object 'withProperColumnNames.<locals>.helper'

>>16056 The SO solution is overkill. Try this solution first, to see if it works for your use case: >>> import pickle >>> def f (): ... print ("lain") ... >>> f () lain >>> b = pickle.dumps (f) >>> pickle.loads (b) <function f at 0x7f00c715f730> >>> pickle.loads (b) () lain >>> def deco (fun): ... def ret (): ... print ("decorated") ... fun () ... return ret ... >>> @ deco ... def f (): ... print ("lain") ... >>> f () decorated lain >>> b = pickle.dumps (f) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: Can't pickle local object 'deco.<locals>.ret' >>> import functools >>> def deco (fun): ... @ functools.wraps (fun) ... def ret (): ... print ("decorated") ... fun () ... return ret ... >>> @ deco ... def f (): ... print ("lain") ... >>> f () decorated lain >>> b = pickle.dumps (f) >>> pickle.loads (b) () decorated lain >>> If it does work for your use case, I'll write out an explanation as to why.

This is a silly question, but have any of you heard of an "uwufier" it subsitutes certain letters to make everything sound like it's being spoken by a japanese cat-person. I was wondering how y'alls might approach that, and if you know where I could find the original project? It was posted on 4chan/g/, and I thought the idea was hilarious but i didn't have the time to try writing it at the time, and now I don't have the description for it.

I kind of want to write an imageboard scraper using Python but damn do I suck at database related stuff.

>>16244 why database? You can serialize objects and save them to files

Sorry for asking this type of question but is Python good to start programming? I'm try Portugo that is a language to learn the programming logic but I think it's very boring and I wish I could start programming in Python because people say that is a good language to begin.

>>16251 It depends what are you trying to achieve. In the long term for first prog lang i recommend Processing language (which in deed is a java library in the underground). Its very nice because is graphical centered language and after using it for a while you will be able to understand java, its an abstraction layer very nice for begginners and easely you can test things and see results. (PLUS: processing code just can be converted to android proyect, but impossible to launch iOS version by using it). If you are looking for OOP (Object Oriented lang) then (for long term) i recommend you to start with java or c++ because they put the bases for all other OOP langs. After learning any of them you will be ok for any other OOP lang. Python its really nice, give it a try, if it breaks your head go Processing for a while and make some schway gifs about Lain, lainon. https://processing.org/ with all my love and bad english. tn13

>>16252 >>16253 Thanks, I wil try these languages and I didn't even know these languages. Btw, I will keep my work learning Portugol because is a programming language that is made to people that speaks portuguese learn how to programm and I'm finally understanding the concepts that are essential to use in Python or C, for example. Also, don't mind the bad english, i'm also really fuarrrked up in terms of knowing english. Btw, thanks a lot, lainon!

>>16251 I don't think you should worry if it's in portuguse or not. Most languages appear to be English but they are just derived. Var is short for variable, Func for function. Either way we all learn new vocabulary. Also processing has been ported to other languages. So you might be able to use it in your desired language.

>>16257 I see, the problem to me was that I didn't see any sense in Portugol or any system that I could create with the language. But even it being boring to use to program it made me learn the basic concepts of the logic of programming.

So I'm working through a textbook on python: There's this question which asks you to write a program which prints a roman numeral for the numbers 1 to 10, It's supposed to be answered with a long list of nested if-else statements, but I was wondering what the actual way you would approach that is like <code> number = int(input('Give me a number: ')) for number: print(numeral) # and then you'd declare your list of numerals I = 1, II = 2, etc. </code> I'm sure there's a real straightforward way of doing this which doesn't require 9 nested else statements, so if someone could let me know, just for my own curiosity/sanity I'd really appreciate it.

Does anyone have a book about Python 3 for aggressive InfoSec? Been reading Black Hat Python, but it uses Python 2 and, to be honest, it isn't a really good book.

>>16365 I would make a list of numerals 1-10, and then when someone checks for number 6 it pulls out numerals[6-1].

>>16365 Well, if it's just 10 numbers then you can simply store them in a list. x = ["I", "II", "III", "IV", "V"] #etc. print(x[int(input())-1]) It might be useful to notice, that you can't have three same consecutive characters. Use your data structures to come up with a solution for any number.

>>16371 Thanks for your reply, so it looks like #declaring numeral list >x = ["I", "II", "III", "IV", "V"] #print list item for given int >print(x[int(input())-1]) Is input() a function? And what is the -1 at the end doing? I realise these are pretty newbie questions but I appreciate the benefit of your knowledge/feedback. >just checked the textbook and lists aren't covered for another 4 chapters I'm making slow and steady progress though.

Python 3 doesn't natively support POSIX so I don't support Python 3.

>>16391 >Is input() a function? It's the same as >number = int(input('Give me a number: ')) in your post. It is a built-in function. >And what is the -1 at the end doing? Lists are indexed from 0 (x[0] is the first element), that's why we have to subtract 1 to access the element we want. >I realise these are pretty newbie questions I suggest you read through the docs on python website, especially the tutorial: https://docs.python.org/3/tutorial/index.html

>>16396 also the int() changes the input to type integer, because input probably returns type string, obviously if someone inputs some characters it won't be able to convert words to numbers and the function will fail.

>>16396 >>16398 Thanks again, I'll go read this.

>>16263 Hello Br

>>16053

I started a few days ago with Programming with Python 3 and I feel like a massive fuarrrking retard because my programs are nowhere as good as the ones in the book. How could I read a string and discriminate which commas are needed and which are not in a CSV file? e.g: COUNTRY,2000,2001,2002,2003 PAKISTAN,"5,6",10,"24,5",7 If I split the strings by , or " everything goes to soykaf, I don't know how to discern which commas are floats.

>>16948 >>16948 it is not the easiest task. it's easy to hack it together, and that's how a lot of people do it, but in general this is a parsing problem, and while well understood, it requires significant background, unrelated to python. if i were do it, in the most naive way possible, i'd "walk" the string with find method. p=0 while True: i = s.find(',', start=p) if i == -1: break … p = i + 1 you maintain your state in variables outside of while loop, like "currentRow=[]" for current row and "rows=[]" for the whole thing. you hack various checks in, like if s[p] == '"', then instead of searching for comma, you search for a closing quote. p=0 while True: if p[0] == '"': i = s.find('"', start=p+1) else: i = s.find(',', start=p+1) … writing something like that is fundamental algorithmic thinking and teaches you methodical approach where you keep state in a handful of variables and do careful case analysis. you explicitly want to avoid high level functions like string.split until your basic algorithmic thinking is together.

>>16949 That was really helpful, thank you! >you explicitly want to avoid high level functions like string.split until your basic algorithmic thinking is together. Do you think I should drop as many built-in functions as I can? The most engaging thing has been making small algorithms like the sorting one.

>>16950 >Do you think I should drop as many built-in functions as I can? The most engaging thing has been making small algorithms like the sorting one. That's not exactly what lainon is saying. Generally (unless you think you can do better, are reimplementing something for learning's sake, or whatever) you want to use the good stuff that's in libraries. Hell, Python got popular on its reputation of being "batteries included". It's just that in this case, string.split is the wrong tool for the job.

>>16950 >>16951 what i'm saying is that for didactic purposes it's useful to be weary of builtin functions. i wouldn't make it a hard rule though, because i don't know where anon is in his learning. if you're using python as a way to learn programming in general, then python's builtin's are going to shape the way you approach algorithms. a lot of criticism of "perl style" programming was about this very issue: instead of being able to approach a problem from first principles, a stereotypical perlist will just hack his way through the problem by using a handful of broad builtins. using string.split(',') to parse a csv is a classical mistake of that nature. i think it's a fundamental programming skill to be able to approach a problem with a handful of constructs in a turing complete language: add two numbers together, get a value of character in string, move things around in a fixed length array, that sort of stuff. when all else fails, you always should be able to solve a problem that way.

>>16950 Use as many built-in functions and libraries as you can. They're almost always going to be faster, more robust, more stable. The catch is that you have to understand what these things do, before you're gonna be able to use them effectively. For your problem specifically I would have gone online and typed something like "CSV parse library python" into a search engine, maybe restricting the results to first-party python documentation before going and looking for third-party libraries. Then your program would probably look something like CSV.parse(...) Sometimes I wish I was born in the 1970s, but mostly I'm glad I wasn't.

>>16955 that's why i don't like chan approach to education, because there's always a dozen of "experts" providing contradictory solutions. it's quite clear from the description anon gave that he's trying to solve a from scratch problem while learning how to program. the answer i gave was fine, but no, everyone needs to demonstrate their expertise. in python you parse csv with builtin 'csv' module, and the top level entry function is 'reader', so it's import csv csv.reader(...) that's literally five seconds to lookup in the documentation. > context of your question doesn't mater, use everything in the standard > faster! > google for answers! the rest of your response is equally lazy.

>>16956 I think it's important for lain to learn how to differentiate between work he or she has to do, and work that's already been done for him or her

>>16957 Obviously you learn to program by solving problems. You should like never use your own implementations of standard algorthims in production code - at least if you can avoid it, but need to implement them to learn.

This might look like a rant, I am angry an ranty cause covid lockdown. But I am honestly asking a question, and I am really looking forward for an explanation of something that changes my mind. I am not a developer, but a data scientist, I have no formal education on software engineering, there is lots of stuff I dont know. So, I have a project, data science stuff, random forest, blah blah blah. Project is 3 .py, one processes the data, one trains the model, one puts a pretty output for the world to consume. There is an actual developer in my team, so he is in charge of writting an API for everyone to consume. The features of the API are: if given a json, predict the model, if given a product id, go to database, retrieve json, predict the model. So now my 3 .py project is an 8 folder project with, with 3 .py per folder, each with 200-300 lines of code. It just feels extremely bloated to me, poorly designed. Now, they are using swagger to generate all that stuff, and swagger should be great, but I just cannot see the point. Furthermore, when I have developed python with swagger, I felt like witting a compiled language, you go, do some changes, mostly not in yaml, rerun the swagger thing, spend more time fitting your own code in the new project's strucutre... Hated it. But it is our official tool at work, so I kinda need to love it.

>>18105 >But I am honestly asking a question If only there was a symbol that indicated whether a question is being asked or not. I propose we use the exclamation mark, but bend it slightly to indicate that it's an interrogative sentence and not to show emphasis. To answer your... rant: Designing APIs that are extensible sometimes takes code - it's a reality. I'm sure you've came across libraries such as NumPy, SciPy, Pandas, Sci-Kit, Tensorflow etc. Guess what, each of these libraries likely has 10s of directories and hundreds of files with thousands of lines of code, you just don't bitch about them because the files reside in a directory you don't look at often. Here's my suggestion to you: Either propose a more elegant way to do things, or ask the only developer in your team to provide the API as a package instead (that you can just import) so your fee fees don't get hurt from seeing some files.

>>18107 >If only there was a symbol that indicated whether a question is being asked or not Absolutely true. I would like to understand the rationale behind the swagger codegen. Where can I get a deeper understanding of swagger and its codegen???. The output feels bloated to me, but is probably designed by people that know a lot more about software than I do, and I have a suspicion that I hate the thing cause I am missing something. Now to answer you, again, with my limited knowledge. The fact that there is bloat out in the world is not an excuse to put more bloat out. If the api need to do 2 things, query database and call a function on the query results, maybe we dont need 20 classes. Maybe we do, but I am missing something. numpy and family have tons of functionality, our api just needs to expose a predict method of a scikit class I have proposed not using swagger, it got turned down, cause swagger is industry standard or something like that. And I need to keep extending the functionality, with the project the developer generated. He needs to call my code, not the other way around. Thank you for your response. Again, I am kinda angry right now, sorry if I am coming through aggressive and ranty, I appreciate your comments.

>>18105 >3 .py project is an 8 folder project with, with 3 .py per folder, each with 200-300 lines of code >I am not a developer, but a data scientist, I have no formal education on software engineering, there is lots of stuff I dont know. It seems to me that smaller files probably indicate more modular code. This is probably a better outcome than a single monolithic file. More files isnt obviously bad, and may indicate better design, but obviously that depends on code in files.

>>18105 I don't know anything about Swagger in particular, but code generation is unusual in Python. Typically you can instead sort it out at runtime and not bother with generating files. So without really knowing what I'm talking about, I think it's one of these things, or multiple: - Documentation. You can easily read the output and see what's happening without digging through abstractions. - Manual tweaks. If the generated code doesn't do what you want you can reach in and change it directly. The danger is that changes are undone when you rerun the code gen. - It matches other languages. Python's runtime tricks don't exist for all languages, but they want a unified approach for all languages they support.

>>18109 Thanks a lot for your comment. I must have miss-explained myself, cause that is not at all how it went. We did not have a project with 6000 loc spread across 3 .py and now we have 30 .py. We had a project with 3 .py, the largest one with 200 loc, implementing all core logic for the api and the whole machine learning process, to having 30 .py with 200 loc each on top of my 3 .py, just to expose the core logic through an api. If I were to do that, rather than 30 .py, it would have been 1, with about 10 lines of code and a bunch of flask boilerplate. This is why I feel swagger codegen is bloated.

>>16179 This looks pretty cool, with real world applications. Thanks!

>>18112 Obviously an order of magnitude of complexity in difference is notable. Is there any improvement in feature set that might explain this? Or perhaps your develop is overcomplicating the project to play with frameworks and tools he wants to learn.

>>18118 Likely, there is a whole model-view-controller pattern implemented, for a project with no views, and barely any controller, just db access. But in terms of business functionality, I wrote a function, that function is exposed through an api using flask. >>18110 I suspect we might not be talking about the same thing. Again, lots of ignorance, but swagger generates code. I might be refering to templating and not codegen. I heard someone call it codegen.

Is there any Python quick reference guide/book, probably only two or three pages long?

>>18133 https://learnxinyminutes.com/docs/python/ https://docs.python.org/3/tutorial/index.html

>>18133 I'm not sure if this is what you mean by quick reference, but I rely on the internal docs a lot. Just type help(whatever) in the repl. I feel like new Python programmers underestimate this just like new C programmers underestimate the man pages. If you're still getting used to syntax stuff and basic language features than the online docs might be better like >>18134 said because they're a bit more "browseable" if you don't know exactly what you're looking for.

sneklang is great and i love it!

I wrote a shunting yard parser (or is it a lexer, I don't really understand the difference) that tokenizes mathematical expressions into a deque consisting of numeric literals and simple arithmetic operators. Please tell me how I could Improve it: python def shunting_yard_parser(string): op_priority = {"/": 2, "*": 2, "+": 1, "-": 1} op_stack = deque() output = deque() # iterate over each token in the string for token in string.split(" "): # if it is a number add it to the output if token.isdigit(): output.append(int(token)) # if it is an operator, push it to the stack elif token in op_priority.keys(): # forall ops >= new op, pop it from the stack & add to output for op in op_stack.copy(): if op_priority.get(op) >= op_priority.get(token): output.append(op_stack.pop()) else: break # push new op to stack op_stack.append(token) else: # raise an error for illegal operators or literals raise SyntaxError("'{0}'".format(token)) from None # add remaining ops to output while len(op_stack) > 0: output.append(op_stack.pop()) return output This is the third time I've ever programmed in my life so I don't know If this is a dumber or smarter solution for my third one-run attempt ever. It took 6 hours to program in total with 1 google search on deque methods. >>18136 this

>>18137 >or is it a lexer, I don't really understand the difference A lexer produces tokens from a character stream. A parser takes those tokens as input and produces a structure that represents the logical relationship between tokens, such as a parse tree or your RPN list. When speaking informally the combination of these two steps is sometimes also referred to as parsing. >Please tell me how I could Improve it There are a few efficiency issues that would improve your code, but what you need to fix first is correctness. >>> shunting_yard_parser ("1 + 4 / 2 / 2 + 1") deque([1, 4, 2, 2, '/', '/', '+', 1, '+']) Your parsing of this test case is wrong and would evaluate to 6 because you are performing the 2/2 division first. The correct parse would output the first division before the second 2, and would yield 3 overall. The problem is your 'for' loop over the copy, which traverses op_stack in queue order instead of stack order. As a result the 'op' in the priority comparison is not the same as the one popped off, because they are at opposite ends. To fix this dump the copy(), switch the 'for' loop to a 'while' loop with the appropriate condition, and peek at the top of the stack using op_stack[-1] as shown in the example in the docs: https://docs.python.org/3/library/collections.html#collections.deque To write your 'while' condition recall that containers have a boolean value that reflects whether they are non-empty, and read the section about the short-circuit nature of boolean operators: https://docs.python.org/3/tutorial/datastructures.html#more-on-conditions

nsa has declassified their python3 training book https://nsa.sfo2.digitaloceanspaces.com/comp3321.pdf

>>18149 Awesome!

>>18149 Can someone upload the pdf here? It seems that I can't view the link because I have js disabled and I'm not willing to enable it.

>>18226 What are you looking for? A story time(it's 326 pages)? Or do you want it adjusted to a different format when you could always right click and save? I'll help you put anon, you just need to say what you need.

>>18234 I'm asking if someone cares to upload the pdf on lainchan so I can view/download it. I can't view the pdf from the link >>18149 provided because I don't want to enable javascript.

>>18226 wget https://nsa.sfo2.digitaloceanspaces.com/comp3321.pdf I think you're JS blocker is fuarrrked, there's no JS on that page it's just a pdf. It's probably blocking your web browsers build in pdf viewer.

Python 2 is out, so there's no need to especially mention "Python 3", the "Python 3" is the Python.

>>18250 Maybe it's being blocked because I'm using it with tor but that doesn't make sense either because I was able to wget it with torsocks...

I've been trying to make a clean socket client for the last few days which sends and gets json objects. I have no fuarrrking idea what I'm doing. If anyone is willing to help I'll be much arigatoed

How to implement pic related? I need to classify prices to fuzzy subsets and assing them parameters according to their price. Two prices being in DeltaPriceMedium should go to the same group MuMediumBuy but with DIFFERENT VALUES. For example: DeltaPrice=150$ should get parameter: 0.86 in MuMediumBuy, but DeltaPrice=290$ should get parameter 0.15 in MuMediumBuy.

>>18285 what do you mean by >clean socket client ? python's standard lib has a json parser. it just converts json to dicts. are you trying to connect to a server over a network? or do you actually want "raw" access to unix sockets? if it's the former, just use the requests library. if its the latter i've never done it but i could probably point you in the right direction if you tell me what you're trying to do.

>>18286 I think you might need to give some more explanation about what you're trying to accomplish.

>>18298 I dont know, all I know is this >The connection is performed by clean socket connection. >The communication protocol of the API uses JSON format. I've tried some other 3 ways to just log in, this is the last one I tried : import socket import ssl import json hostname = 'h0st.com' context = ssl.create_default_context() j_object ={"command": "login","arguments":{"userId": "u53er1d","password": "p455w0rd"}} with socket.create_connection((hostname, p0rt)) as sock: with context.wrap_socket(sock, server_hostname=hostname) as ssock: print(ssock.version()) print(ssock.send(j_object)) print(ssock.recv(j_object)) # wait for two seconds time.sleep(1) # close connection ssock.close()

>>18311 Sending data of sockets is hard because you dont know how much data to read. What you put in send() may show up in many packets and take many recv() requests. If you call recv() at the wrong time, you program will hang. Sockets programming is hard check out the unix philosophy thread for why.

Still rough beginner at serious CS/Python in general (only had experience in basic code noodling years ago). Is this the best way to write a function to return prime factorization? No string formatting so it just spits out the raw factors. I feel like there are more concise/efficient ways to get the same result. python def primeFactor (n): factors = [] notprime = True while notprime: for i in range(2, n): if n % i == 0: #append factor, restart loop with new number n = int(n/i) factors.append(i) break elif i == n-1: #number has no divisors (prime), append number and break loop factors.append(n) notprime = False return factors Maybe I'm overthinking it but I'd like to know your thoughts.

>>18311 - You are very unlikely to want send and recv outside the inner 'with' block. - You'll want to encode j_object in some fashion. - Make sure you are actually required to use sockets directly. In most situations you'll want to post a json to a https url instead. https://docs.python.org/3/library/urllib.request.html#examples >>18314 Check out the // operator, the 'else' clause of 'for' loops, the sqrt function and the sieve of Eratosthenes.

>>18311 Yeah i got it in the inner 'with' block, it was a paste error or something. I guess it's logical to have it in the inner one. >>18312 I'm not launching rockets to the moon. I figured it out, this works with no errors import socket import ssl import json import time hostname = 'host.com' context = ssl.create_default_context() j_login = {"command":"login", "arguments": {"userId":"test", "password":"test"}} j_object = json.dumps(j_login) print(j_object) with socket.create_connection((hostname, port)) as sock: with context.wrap_socket(sock, server_hostname=hostname) as ssock: print(ssock.version()) print(ssock.send(j_object.encode('utf-8'))) print(ssock.recv(1024)) # wait for two seconds time.sleep(1) # close connection ssock.close() Thanks a lot for helping out!

>>16179 So, I really like this book so far, but I ran into an issue about 131 pages through. If you look at the attached images: >He says to add a line to the code >it gives a traceback error >I think,"OH I must have to convert it to a string!" >Doesn't work, still adds the value by two >"Well maybe if I loop it >more traceback errors, now saying that "x_increment" is not defined >fuarrrk it back to formula >still not working >figure, "What if I copy paste the whole program?" >wtf it actually works? This can't just be because I'm new to coding, right? There has to be a more efficient way to add values then just copy pasting a whole if-elif-else block. I know it rudimentary, but does anyone have any ideas?

>>18337 continued

>>18337 >>18338 I can't help much, because I don't know exactly what you're trying to do. I'll try anyway. >He says to add a line to the code >it gives a traceback error >I think,"OH I must have to convert it to a string!" Yep, that looks like a mistake in the book. Good spotting. alien_0['speed'] = fast Should instead be alien_0['speed'] = "fast" >Doesn't work, still adds the value by two Yes. You're changing the speed after the alien's moved. To get the alien to move three steps, you would need to put alien_0['speed'] = "fast" between creating the alien and moving it. >figure, "What if I copy paste the whole program?" >wtf it actually works? >There has to be a more efficient way to add values then just copy pasting a whole if-elif-else block. I'm not entirely clear here. Are you trying to move the alien twice, at two different speeds? If so: Your intuition is exactly correct. Copying and pasting the program will work, but it's obviously not the "right" solution. If you're working through that book from start-to-end, you haven't been introduced to functions yet - they're in Chapter 8. Skipping ahead a bit, what you want looks like this: def move_alien_right(alien): if alien['speed'] == 'slow': x_increment = 1 elif alien['speed'] == 'medium': x_increment = 2 else: x_increment = 3 alien['x_position'] += x_increment alien_0 = {'x_position': 0, 'y_position': 25, 'speed': 'medium'} print("Original x-position: " + str(alien_0['x_position'])) move_alien_right(alien_0) print("next x-position: " + str(alien_0['x_position'])) alien_0["speed"] = "fast" move_alien_right(alien_0) print("final x-position: " + str(alien_0['x_position'])) That way, you can re-use the if alien['speed'] ... code as many times as you want.

How about a configuration dict mapping speed strings to increments?

>>18342 >I can't help much, because I don't know exactly what you're trying to do. I'll try anyway. You hit it right on the head. Thank you so much anon. I understand it a bit better now.

>>18342 is there a difference between " and ' in python? i've always used them interchangeably.

>>18350 ' is sometimes used in words like "Sheila's" to show ownership, but python would get confused if you used it like 'Sheila's workshop' in a string. That's the only reason you'd use " over ', to differentiate. So no, there's no difference, only times where you can't use '.

>>18350 https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

>>18310 its fuzzy logic problem, look here https://pythonhosted.org/scikit-fuzzy/auto_examples/plot_tipping_problem_newapi.html

>>16053 Hello Lain, I wrote a short textbook on Python recently with absolute beginners in mind, although I have been told it is useful to a season programmer who has not learned Python. >inb4 don't dox yourself I'll take the risk. I hope this helps somebody.

def bubblesort(l): for i in range(len(l)-1): for i, e in enumerate(l): if i < len(l)-1: #Don't go out of bounds (index errors) if e > l[i+1]: l.insert(i, l[i+1]) #swap location of smaller number l.pop(i+2) #delete old location of number return(l) I'm gonna start working through some algorithms and implementing them into python. I learned the concept of a bubble sort and worked this out. Does this look good?

>>18467 Thank you! I'll let you know how it goes

>>18795 >Does this look good? - Reusing the 'for' index is a poor idea for readability. - Len(l) is cheap but not free. - Use sane swaps by assignment instead of repeatedly shifting everythng. - Dump enumerate and use a range to get rid of the first 'if'. - Decrease the upper limit of the inner loop by one on each outer step, as the highest element is guaranteed to get to the top in a single round. - You may further narrow the limits of the inner loop more aggressively by taking advantage of the lowest and highest recorded swap locations.

>>18795 Or you could just user the .sort built-in written in C by more clever people.

>>20163 You obviously don't like learning

>>18795 >>18799 >- Dump enumerate and use a range to get rid of the first 'if'. This is probably the biggest one for me. If you were describing the bubble sort algorithm to someone, the 1st 'if' statement would probably not come up. This is a sign that there's probably a better way to frame the problem, particularly in python. >- Reusing the 'for' index is a poor idea for readability. Ideally, the indices would have some meaningful name. Maybe the first index could be pass_number or something. - Use sane swaps by assignment instead of repeatedly shifting everythng. This is problem for readability and efficiency. The pythonic way would be like: list[i:i+2] = reversed(list[i:i+2]) > return(l) I'd add that it's a bit strange to return from a method that modifies in place, but it could be convenient. >- Len(l) is cheap but not free. >- Decrease the upper limit of the inner loop by one on each outer step, as the highest element is guaranteed to get to the top in a single round. >- You may further narrow the limits of the inner loop more aggressively by taking advantage of the lowest and highest recorded swap locations. These are performance optimizations that probably would detract from readability. I would say the code looks perfectly good without these changes, but if you are trying to write the most efficient code at expense of readability, these changes would make it better. >>20163 The people who wrote it might not have been more clever. >>20167 or maybe they do like learning, but they just misunderstood the purpose of the post they were replying to.

>>20269 >list[i:i+2] = reversed(list[i:i+2]) This constructs the slice, then constructs the reversed slice, then performs slice assignment i.e. a loop. Use either three assignments with a temporary or better yet p,q=q,p to avoid all the intermediate objects.

>>20273 If you're trying to learn how to write super-efficient code, i would suggest writing in another language, such as C++. When writing real code, you should only optimize something if you are sure it is a large contributor to your code's slowness.

>>20401 1. Someone writing bubble sort is obviously learning algorithms, not "writing real code". 2. There is a difference between normal to efficient and wasteful to normal. Eliminating the reversed slice assignment is in the latter category. 3. Eliminating small, temporary, needless allocations in your innermost loop applies to any language. Not doing so is in the class of gross inefficiencies sometimes known as memory burn.

>>20402 >1. Someone writing bubble sort is obviously learning algorithms, not "writing real code". Idk, maybe they just wanna learn python and algorithms were the first thing they thought of.. It's good to point out the positives and negatives of different approaches though.

I'm very new to programming in general, never had any experience before and decide to start with Python due to recommendation. I decided to make a generator of Cyberpunk 2020 lifepaths and now I'm stuck, help me understand the For in Loop, already searched it but I'm too dumb to understand how to apply correctly. import random d10=random.randint(1,10) def siblings(roll11): def gender(roll12): if (roll12 % 2)==0: return 'Sibling gender: Female' else: return 'Sibling gender: Male' def agedifference(roll13): if roll13<6: return 'Sibling relative to yourself: Older' elif roll13==10: return 'Sibling relative to yourself: Twin' else: return 'Sibling relative to yourself: Younger' def feelings(roll14): if roll14<3: return 'Sibling feelings about you: Sibling dislikes you' elif roll14==3 or 4: return 'Sibling feelings about you: Sibling likes you' elif roll14==5 or 6: return 'Sibling feelings about you: Sibling are neutral' elif roll14==7 or 8: return 'Sibling feelings about you: They hero worship you' else: return 'Sibling feelings about you: They hate you' if roll11>7: return 'Sibling: You are a only child' else: for i in range(roll11): gender(d10) agedifference(d10) feelings(d10) j=siblings(d10) print(j) Why it doesn't work? When the roll is 9 or 10 it prints "Sibling: You are a only child" like it's supposed to, but anything else it prints 'None". Pic is the logic I'm trying to replicate.

>>20605 The first major problem I see is your use of "or". The way or works is you have STATEMENT OR STATEMENT Where STATEMENT needs to evaluate to a boolean of some kind. Though it works in English, you cannot say "roll14 == (5 or 6)", which is what you think you are asking. You are asking "roll14==5 OR 6", and 6 is going to evaluate to True because it's a legacy from C that 0 = False and any value = True. Your second issue, and the cause of your problems, is that you're not returning anything from siblings(). Why? Let's trace the code. You call siblings(). Inside of siblings() you define several functions. Then, you check if you're an only child and return a string. This string is then assigned to j and the program exits. If you are not an only child, it will start a loop which will run between 1 and 10 iterations, rolling a d10 each time. First it will call into gender(), which will take a randomized value and then do an even/odd comparison. This function will return a string. However, when you get back to line 37 and return the string, it is not saved anywhere. And this repeats for the following functions. They return a value to inside of siblings(), but nothing is saved. The loop runs n times and then exits. By default, when a function ends, it returns. If no explicit return value is defined then Python will return None. This is equivalent to a void function in some other languages. You can see this by adding a return statement to your gender(), agedifference(), and feelings() calls. Except now it will just print the gender. Why? Because you returned what gender returned up another level, but return always exits the function. The quick fix for the way you've written this is to make an empty string s = '' and then concatenate the return values. When done, return the string. There are more optimizations you could do, and you should switch to using camelCase for your function names to make them more readable. But given your architecture, appending to and then returning a string would be the best match for the way you've written your function. Please let me know if you need further clarification.

>>20610 >you should switch to using camelCase for your function names to make them more readable Not anymore, PEP8 suggests lower_case_with_underscores over the old mixedCase. That's also what you'll see in modern python projects. >Function names should be lowercase, with words separated by underscores as necessary to improve readability. >Variable names follow the same convention as function names. https://www.python.org/dev/peps/pep-0008/

>>20610 >If you are not an only child, it will start a loop which will run between 1 and 10 iterations, rolling a d10 each time. There is no rolling, Anon's d10 is computed exactly once. >>20605 In addition to >>20610 you might also want to actually roll. def roll (upto): return random.randint (1, upto) Then use roll (10) as the argument to the invocation of your four functions.

>>20616 >spaces before parentheses

>>20613 PEP8 is great but I think it leaves a little to be desired. I've universally switched to using snake_case for variable names, camelCase for functions, CamelCaps for classes, and dashed-words for anything executable. This is not PEP8 compliant but it provides greater clarity regarding what you are looking at. Most of PEP8 is solid though and I strongly recommend people use a PEP8-compliant linter when writing Python. It will help enforce some of the whitespace quirks. >>20616 Good catch. He could also use python import partial roll = partial(random.randint, 1) Likewise, roll(10) would then roll a d10.

>>20619 >import partial Perhaps you meant https://docs.python.org/3/library/functools.html#functools.partial since: $ python3 [...] >>> import partial Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'partial'

>>20610 I do need further clarification, I applied what you said like this def siblings(roll11): def gender(roll12): if (roll12 % 2)==0: return 'Sibling gender: Female. ' else: return 'Sibling gender: Male. ' def age_difference(roll13): if roll13<6: return 'Sibling relative to yourself: Older. ' elif roll13==10: return 'Sibling relative to yourself: Twin. ' else: return 'Sibling relative to yourself: Younger. ' def feelings(roll14): if roll14<3: return 'Sibling feelings about you: Sibling dislikes you| ' elif roll14==3: return 'Sibling feelings about you: Sibling likes you.| ' elif roll14==4: return 'Sibling feelings about you: Sibling likes you.| ' elif roll14==5: return 'Sibling feelings about you: Sibling are neutral.| ' elif roll14==6: return 'Sibling feelings about you: Sibling are neutral.| ' elif roll14==7: return 'Sibling feelings about you: They hero worship you.| ' elif roll14==8: return 'Sibling feelings about you: They hero worship you.| ' else: return 'Sibling feelings about you: They hate you.| ' if roll11>7: return 'Sibling: You are a only child' else: for i in range(roll11): s = '' s = gender(d10) + age_difference(d10) + feelings(d10) return s But I don't think that's what you meant, I am indeed getting al the three, but only once instead of the multiple (random but bellow 7) times. >>20616 >>20619 What's the advantage of doing it like that? I know I can change more easily the value of the dice, but it's not something I plan to change later.

>>20624 - You are not rolling, you are using the same d10 value for everything. >>20616 - Move the 's' initialization before the loop. - Move the 'return s' after the loop. - Switch the 's' update to concatenation: 's += gender ...'

>>20624 elif (roll14 == 3) or (roll14 == 4):

>>20629 Oh, I see. Thank you (and everyone else) so much for the help, it's working perfectly now. I'll have to change the d10 in the rest of the file too because I've been using it wrong since the start but now I'm not stuck anymore! >>20630 I see, I'm even dumber than I thought before but that's fine, at least NOW I think I understand the or statement.

>>20605 >>20639 Disregard this post if it confuses you, for it doesn't contain Python advice but pointless daydreaming and foolish hackery. With one exception: in your case, I'd write if roll14 in [3, 4] but only out of stylistic preference, there's nothing wrong at all with the previous suggestion. On to the nonsense now. It's been explained why "roll14==3 or 4" doesn't do what you mean, but I ask: why shouldn't it? Indeed, as already admitted, "if roll14 equals 3 or 4" is almost unmatchably intuitive and readable, so why shouldn't a language support it? The obvious problem of ambiguity with Boolean or, and the slightly subtler implicit limitation to two arguments that "or" imposes, can be solved at once by choosing another word, like "either". Consider these crude examples: 7 == Either(8, "somestring", 7, None) # => True 9 == Either(5, 3) # => False Are they not readable and effective at conveying intent? They can be implemented with as little code as: python class Either: def __init__(self, *values): self.values = values def __eq__(self, other): for val in self.values: if val == other: return True return False This is a poor hack, as it covers only the case of equality comparison and not the broad range of Boolean option-picking that either should support. I have some slightly less bad ideas, but it may well be the case that Python is the wrong language for this kind of pursuit - it doesn't seem to fit its spirit. Whatever the case, I've been positively inspired and I wanted to share these thoughts and thank lainon for his inspiration.

>>20644 This Either is i think damned to forever be a hack because it won't be able to handle arbitrary expressions. "If the type of arg1 or arg2 is int" probably can't work. I would suggest one do it the "one obvious way" that python provides: any(map( lambda x:7==x, (8, "somestring", 7, None) )) any(map( lambda x:type(x)==int, (arg1,arg2) )) It might seem cumbersome at first, but i think once you get used to it it's as natural as if roll14 in [3, 4] , and it scales to situations where "or" would get confusing in English.

>>20648 We think alike. Indeed, one of my less terrible ideas was the realisation that the barebones either is simply a special case of any applied to the identity function. But we disagree in one aspect: lambdas are certainly convenient and easy to get used to, but never, I think, quick to read. This isn't a Python problem - from Scheme's wordiness to Ruby's streamlined "->{}", procedure literals require a few ticks to parse and reason about. What I'm after is readability in the very demanding sense of reading pretty much as if it were English - surely we can get closer to that? The lambda problem for one can be solved trivially by defining predicate generators such as: python def equals(n): return lambda x: x==n def is_type(t): return lambda x: type(x) == t Now we can ask questions like: any(map( equals(7), (8, "somestring", 7, None) )) any(map( is_type(int), (None, "not me!") )) And that's better, but the nested function calls are still distracting. We can clean up the syntax with some sugar: python def either(*args, does=lambda x:x): return any( map( does, args ) ) And now, if you'll only allow me to rename the above predicate generators to accommodate for English grammar, we can ask these questions: either([], None) # => False either(8, "somestring", 7, None, does=equal(7)) # => True either(None, 7, "not me!", does=have_type(int)) # => True And now I'm happy. Do note that I'm insisting on the name either only to avoid stepping on the built-in any.

>>20648 Why are you map'ing? You can use pythons generator syntax directly in any parentheses, even ones that call a function: any(7 == x for x in (8, "somestring", 7, None)) any(type(x) == int for x in (arg1, arg2))

I'm fuarrrking around with Tkinter and i wanna remove the title bar from it overridedirect removes everything and i don't feel like rebuilding everything from scratch Anyone have any ideas?

>>>[f() for f in [lambda: x for x in range(10)]] [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] What is this soykaf ?

>>20702 By the time you call the lambdas, x is 9. This is unintuitive but not a great mystery.

>>20703 Do note, however, [f() for f in [lambda x=x: x for x in range(10)]] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>20702 That is one of the items in the Programming FAQ: https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result """Note that this behaviour is not peculiar to lambdas, but applies to regular functions too.""" Lambdas https://docs.python.org/3/reference/expressions.html#lambda Function definitions https://docs.python.org/3/reference/compound_stmts.html#function Naming and binding https://docs.python.org/3/reference/executionmodel.html#naming

>>20702 >>20705 I get it, first one X is like a global variable, second one the Xs are copied as local variables/constants.

Thank you. I found the underlying reason here: https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture > Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it's pointing to. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

>>20705 Maybe stop wasting time fuarrrking around with riddles and just write what you need to something useful.

>>16948 if you just want to figure out how to read a csv file, i suggest doing it line-by-line and not use string splits at all if you just want python to read a csv file for data manipulation, just pip install pandas and use pandas.read_csv

>>16948 With Pessimism. Assume that nothing you write will ever be good enough, and go find a library that already implements this for you. I personally like to do "garbage in, garbage out" where I don't even look at the fuarrrking data. I just pass everything around as binary blobs and hope that eventually it ends up in front of a human, or other computer system capable of understanding it.

>>20757 Or just use the built-in csv module. Correctly parsing a csv if you don't know how to do it is non-trivial.

>>20748 Why shouldn't i play around with riddles? Anyway, this property of can be very useful.

I'm hoping that lain can help me. I am using multiprocessing to speed up some xxhashing. I have 10s of TiB to get through so every little optimization counts. Right now my code works fine, but there's an annoying bottleneck, where a long-running process can hold up my script, and prevent it from spawning more processes. Here's the code to give a better idea what I am talking about. python import multiprocessing as mp openFiles = 0 processes = [] for f in filenames: if openFiles < 500: openFiles = openFiles + 1 p = mp.Process(target=hashFunction, args=[f]) p.start() processes.append(p) else: for p in processes: p.join() processes = [] openFiles = 0 for p in processes: p.join() So you can see I set a limit on the number of processes which my script will spawn. This prevents my child processes from growing out of RAM and crashing my computer, but it also means that all 500 processes must finish before more can be created. Is there some kind of event handler I can set maybe that could tell me when any process finishes, so I can start up another one? So far I only know how to use Process.join() to wait for specific processes, but that's no help because I have no way of knowing which process is going to finish first, and even if I just pick a random one to join(), I still won't know how many other processes finished, and thus won't know how many more processes I can safely start without running out of resources. Any ideas to keep the number of running processes hovering around 500, for the duration of the computation?

>>20818 This is what ProcessPoolExecutor is for, but 500 is a bit ridiculous. https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example

>>20818 easiest solution would be to just add a timeout to the join call. that way at least the loop doesn't block until a particular thread returns. still not super efficient since the main thread is still going to be somewhat bottlenecked. the actual solution to this problem is to have the worker threads fetch new data and handle output on their own without coordinating with the main thread. you don't even need to coordinate between threads with shared memory if you design things properly. for instance, if you have 20 threads, just give each thread an incremental offset on creation and have them automatically hashes every (20n + offset) data block for n=1,2,3,... This way no two workers will ever try to hash the same block, and the program can simply exit/continue when all threads complete their assigned work. also, there is limited value in creating so many threads. it will probably slow things down, since thread creation is usually an expensive operation. having fewer threads (roughly as many threads as you have cores) that run for a long time is preferable to many short-lived threads.

>>20836 >>20818 basically, something like this (untested): python processes = [] number_of_threads = 16 file_names = ['file1', 'file2'] #etc. def worker_thread(file_names, offset): for file_name in file_names[slice(offset, len(file_names), number_of_threads)]: with open(file_name, 'r') as f: # do your hash logic here if __name__ == '__main__': for n in range(number_of_threads): p = mp.Process(target=worker_thread, args=[file_names, n]) p.start() processes.append(p) for p in processes: p.join()

>>20829 I have a very nice processor.

>>20837 I implemented your suggestion with Pool.map which is working very nicely. Now I just need to make sure file sizes are evenly distributed across my input lists. Such is necessary because if some lists have more big files than other lists, the lists containing small files will finish first, resulting in another bottleneck similar to the one I began with. Also bumped down the number of Pools to match my cores, because it turns out my processor isn't THAT nice. Will post full code later, as I think it could be worth reusing. Also, I am multiprocessing, not multithreading. Python threads are subject to global interpreter lock, and cannot run on multiple cores. >>20829 I think concurrent.futures is an async library? I'm not sure this will provide the performance I need, as my computation does not involve network I/O.

>>20840 >Now I just need to make sure file sizes are evenly distributed across my input lists. Honestly it might not be worth the complexity or (admittedly marginal) computational cost of reading in a bunch of file sizes and sorting them. Since you're interleaving the files across threads you won't end up in a situation where e.g. the first thread gets a bunch of really large files from a single directory. I would try profiling it first with some subset of your data to see if the distribution of file sizes across the subprocesses is actually biased, because a load balancing implementation isn't going to be pretty. Doubly so if you want the threads to load balance dynamically rather than just sorting the list of files before they're spawned. Seems like the need to involve shared memory (and consequently, locks) in such an implementation would negate any benefits you might be able to get. >Python threads are subject to global interpreter lock, and cannot run on multiple cores. Hey, thanks you taught me something!

>>20841 I will say, if you want to do the dynamic load balancing thing you should probably go with a worker queue type architecture instead of the example I posted. That way the main process can load balance and the subprocesses can grab things off the queue instead of having to coordinate with eachother.

>>20841 the load balancer was surprisingly easy to write: python import os processors = 12 def sortKey(e): return e[0] unhashedFiles = [(os.stat(f).st_size, f) for f in unhashedFiles] unhashedFiles.sort(reverse=True, key=sortKey) fileQueues = [[] for i in range(processors)] totals = list(map(sum,fileQueues)) for f in fileQueues: fileQueues[totals.index(min(totals))].append(f[1]) totals[totals.index(min(totals))] += f[0] for t in totals: print(t) This is just a heuristic, and I had to tweak the variables a few times, but I was able to get the loads to match up to like, 9 decimal places for arbitrary number of fileQueues. And now that I'm running it all. It turns out it's faster with no parallelization at all! XXHASH prefers to have the entire CPU all to itself! I'VE WASTED MY TIME! FUUCCKK

>>20844 Oh. I made a typo. Corrected: python import os processors = 12 def sortKey(e): return e[0] unhashedFiles = [(os.stat(f).st_size, f) for f in unhashedFiles] unhashedFiles.sort(reverse=True, key=sortKey) fileQueues = [[] for i in range(processors)] totals = list(map(sum,fileQueues)) for f in unhashedFiles: fileQueues[totals.index(min(totals))].append(f[1]) totals[totals.index(min(totals))] += f[0] for t in totals: print(t)

Where can i start programming at all? I don't just mean learning python. I mean in general, the very basics to programming.

>>20845 some critiques: your sortKey function is not needed. sort() will already sort by the first element of the tuple. this is almost certainly faster than employing a function/closure/lambda. >totals = list(map(sum,fileQueues)) this creates an array of zeros. was that your intention? if so, that's a really strange way to do so. totals = [0] * processors is generally how you would do that. you don't actually need multiple queues. your current approach stores several huge arrays with entries for every single file in memory, which may or may not be a problem, but doesn't scale up indefinitely. there will be a limit to how many files you can handle with your script. a better approach might be to create just one "work queue" (using the thread safe queue object from the multiprocessing library) and fill it with "jobs". workers grab a new job off the queue when they finish their current job so load balancing is automatic. again, untested: python import os from multiprocessing import Process, Queue unhashedFiles = ['some', 'list', 'whatever'] processors = 12 job_size = 100 #number of files per job work_queue = Queue() def main_thread(): working = True while working: job = [] for i in range(job_size): try: f = unhashedFiles.pop() #you could also just slice unhashedFiles into equal job_size batches but this way #it's easier to replace the pop() call with something that e.g. reads a line #from a file containing a list of filenames job.append(f) except IndexError: #this happens when you run out of things to pop() working = False break work_queue.put(job) def worker_thread(): while True: job = work_queue.get() for f in job: hash_file(f) #your hash logic here work_queue.task_done() #task_done() tells the main thread you've finished the job #when all tasks added to the queue are done, this thread can be joined #(just look up how mp queues work) if __name__ == '__main__': processes = [mp.Process(target=worker_thread).start() for i in range(processors)] main_thread() for p in processes: p.join()

>>20849 >processes = [mp.Process(target=worker_thread).start() for i in range(processors)] whoops i just realized this wont work. i don't think start() returns a process object. (also the imports are different in this example, so the mp shouldn't be there). replace it with python if __name__ == '__main__': processes = [Process(target=worker_thread) for i in range(processors)] for p in processes: p.start() main_thread() for p in processes: p.join()

>>20848 there are better threads to ask this question (if there isn't a "how 2 code" thread just post in the programming questions general). but essentially i'd recommend starting with basic control flow. if/then conditionals, for/while loops, etc. then learn how functions work. these are common to nearly every language. then learn how to create custom data types (usually called "classes" or "structs"). the way this is done varies by language, but basic composition (i.e. you can put multiple objects together to make a bigger object) is a very general concept. finally, you can get into stuff that is more language-specific. how are different behaviors/objects encapsulated? in C++ you'd write a class. in python you'd write a module. in rust, you'd write traits and apply them to structs. in C you'd define a struct and write helper functions to manipulate it. python is fairly easy to write, so it's commonly recommended as a first language. i don't necessarily disagree with this, but you should know that some idioms that are very common in python are rarely used in other languages. list comprehensions, for example. whenever you write a `for` loop in python, you should probably ask yourself if it could be done in a comprehension instead. e.g. mydict { key:value for k,v in zip(list_a, list_b) } to create a dict with keys from list_a and values from list_b. In another language, this might be written like map<string, string> mydict = {} for(int i=0; i<min(list_a.length(), list_b.length()); i++) { mydict[list_a[i]] = list_b[i]; } which also tends to be the way new python programmers write things.

>>20853 >mydict { key:value for k,v in zip(list_a, list_b) } sorry, i meant mydict = { key:value for key,value in zip(list_a, list_b) }

>>20849 >>20850 Your worker_thread() runs forever, as does your p.join().

>>20857 Ah, I misunderstood what the docs were saying. task_done lets things that are blocking on the queue continue, not things that are blocking on a thread join. https://docs.python.org/3.7/library/queue.html Anyway, heres my code rewritten to follow what was done in the example. python from multiprocessing import Process, Queue unhashedFiles = ['some', 'list', 'whatever'] processors = 12 job_size = 100 #number of files per job work_queue = Queue() def main_thread(): working = True while working: job = [] for i in range(job_size): try: f = unhashedFiles.pop() job.append(f) except IndexError: working = False break work_queue.put(job) def worker_thread(): while True: job = work_queue.get() if job is None: break for f in job: hash_file(f) #your hash logic here work_queue.task_done() if __name__ == '__main__': processes = [Process(target=worker_thread) for i in range(processors)] for p in processes: p.start() main_thread() work_queue.join() # wait for the workers to drain the queue for i in range(processors): work_queue.put(None) for p in processes: p.join()

>>20863 I already did this and it ran slower. Maybe with a different hashing algo you can get higher throughput, but with xxhash, I observed that parallelizing this task consistently reduced TOTAL bandwidth by at least 50%. Bandwidth is key. If total bandwidth drops, then that means the computation takes longer to process all files. >totals = list(map(sum,fileQueues)) this is a very lispy way of making a list of 0's. LISP was the first language I ever learned and the old habits die hard. It's kind of like how if you ask an ESL person to count backwards from 30, they'll just reflexively start going in their first language. I usually don't clean such things up until it becomes clear that they're causing a bottleneck and for this computation, the bottleneck is not in the load balancer. Speaking of that load balancer. It can balance inputs of size 10^6 in milliseconds, so I don't worry too much. 10^6 filenames takes up about 330 MiB in RAM. I also read and write to a database stored in a ramdisk, so THAT I/O is as fast as possible. It's okay guys. I have ECCs. I read the files in 10 GiB chunks. Eventually this program will run on a Ryzen threadripper with 128 GiB of RAM so that will be fun to tune up and tinker on. In a single process, on my little-dick 12-core i7 and 32 GiB or RAM it's currently hashing about 1 GiB every 7.4 seconds. I don't think this is going to go faster unless I rewrite the whole thing in C or buy a faster disk. For now, the next step is to use these hashes to do integrity checks and de-duplicate files by copying only the ones with unique hashes over to the backup drive. Then once it's copied I have to calculate the hashes again and compare.

>>20863 Your message passing scheme is now correct in principle. However your program is still wrong because your work_queue sharing is dependent on the multiprocessing start method. If the selected start method creates a fresh python interpreter process -- e.g. spawn, the default on Windows -- your module will be imported and the work_queue = Queue() line will create a separate unique Queue for each worker process, so there will be no communication at all. You must ensure that each worker process has access to exactly the Queue instance of the main process. To see how to do this using Process arguments look at the multiprocessing queue example, instead of the multithreading queue that you linked.

>>20865 so you have to pass the queue as an argument to the worker process? because subprocesses get a full copy of the calling process's stack, whereas threads share some context with the main process? >the default on Windows do you happen to know if this is the case on linux?

>>20848 A good book will get you a long way, especially if you are self learning. If you don't want to throw money into something you don't know you will like, then >Automate the Boring Stuff with Python Is probably a good place to start. Very hands-on type of book where you are doing actually somewhat usefull stuff.

>>20868 >so you have to pass the queue as an argument to the worker process? Yes, for multiprocessing you have to. This is because the start method might create a copy of the main process or it might create a fresh python interpreter process, and your program should work in either scenario. You may also pass queues to threading.Thread objects in the same way simply to maintain uniform treatment. >do you happen to know if this is the case on linux? The current documentation states the Unix default to be the fork method. https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

>>20868 Anything you pass between processes must be 'picklable'. In python, pickle is the tool used to serialize and deserialize data as it passes through the OS layer, being piped between processes. These pipes have significant overhead, so if you have enough RAM, it is better to just start every process with all the data it needs. If your data is too big to fully enumerate at runtime, you can start the process with a lazy list instead.

>>20863 I thought you might be interested to know lainon, that if you balance your processes so that the separate processes are pulling from separate hard drives, you can actually do this computation in parallel without losing bandwidth. I just did a quick benchmark and I was able to read from multiple drives in parallel without reducing total I/O between processes.

>>20883 And the data is in folks. 152,866 files. 20,752 unique hashes. I am amazed.

>>20888 How long did it take?

>>20891 36 hours. I didnt realize I could parallelize across separate physical interfaces until I was on my last drive. If I figured that out earlier it would have been more like 7 hours.

>>20892 Holy soykaf, good work!

what are your thought on an OS based practically entirely on python

>>21131 Sounds like a bad idea. I like Python, but it's a horrible pick for systems programming.

>>21131 It sounds impossible. An OS does things like manage memory that python cant do. The Python Runtime makes syscalls probably through libc , how could you implement thoughs syscalls in python? You might be able to rewrite a large chuck of user space in python though as a project.

>>21134 It sounds stupid but isn't inconceivable, nowadays... https://www.netbsd.org/gallery/presentations/mbalmer/fosdem2012/kernel_mode_lua.pdf Depending on what any proposed scripting module does, any required syscalls would require function bindings. That is correct.

>>21134 You don't necessarily need memory management. There's been operating systems written almost entirely in Lisp and Java, both of which are garbage collected. I still wouldn't write an OS in python though. It's nowhere fast enough for lower-level systems programming, and there's large chunks of it that are dependent on other languages like C.

>>21132 >The basic idea is to have a microkernel mostly based on the Python VM and all other operating system functionality written in Python. Not so much of a great idea but these guys are making an interesting project https://github.com/jtauber/cleese I mean the code is 97% C but it's still a nice concept

What do you people use for responsive front end for FLASK applications? I need to build some UI and I am very out of tune with the frontend world.

>>21182 Try out vue.js or react.js, vue is comfier in my opinion (but try not to download a soykafload of npm helper libraries along with it)

>>21184 Thanks, I had bootstrap and vue on the list to check out. React, well, I'm weary of anything facebook makes.

>>21187 Bootstrap is great for making responsive layouts and common web components quickly. It's almost fully responsive out of the box. You will still need to write some custom CSS if you want to tweak its default look, but it will save you a lot of time that you would otherwise spend writing CSS from scratch. Being weary of any facebook technology is completely understandable, but react is open source. In a nutshell, it's just a JS library that allows you to separate your UI logic into reusable chunks called components, and manage their state easily. It doesn't talk to facebook servers or something like that, so it should be safe. Still, I recommend that you use vue instead, it can do everything that react can, but in my opinion it's more intuitive and easier to learn.

>>21189 Thank you. I am weary of react because it feels to be like the Django of frontends. it has so many bells and whistles that I don't need, I'd rather learn something new and work more than get lost in the sea of possibilities that such well-formed frameworks provide. That's why I do Flask. I gotta do a lot more that way, and it provides practice. I will check bootstrap & vue. Picked up a course from M. Kennedy about it just now. Cheers.

I'm a beginner at programming, and it is pretty much my first practical program with python. It's a pretty limited scrapper for gelbooru, but I'm not sure if I made the right choices. Are there more efficient way to do it? These while loops make me feel a bit uneasy. python import requests import glob file_list = glob.glob("*.*", recursive=True) tags = input("Entrez vos tags : ").replace(' ', '+') page_number = input("Entrez le numéro de page : ") or '0' search_url = 'http://gelbooru.com/index.php?page=dapi&s=post&q=index&json=1&tags=' + tags + "&pid=" + page_number r_search = requests.get(search_url) r_json = r_search.json() i=0 x=0 #LIMITATION: if the pictures are renamed they'll be downloaded again while i < len(r_json): string_a = r_json[i].get("file_url") string_b = string_a.rsplit('/', 1)[1] if string_b in file_list: r_json[i].pop("file_url") x+=1 i+=1 while x < len(r_json): curr_url = r_json[x].get("file_url") if curr_url != None: test = requests.get(curr_url) if curr_url.find('/'): filename = curr_url.rsplit('/', 1)[1] print("Downloading... " + curr_url ) open(filename, 'wb').write(test.content) x+=1 print("Done...")

>>21555 There's no point in doing the while loop twice. Just pull the URL out of the json object, check if you've downloaded it already, and if not download it. Also, getting a list of all filenames in any recursive subdirectory and then checking if your file is in that list is probably not the best way to solve that problem. If you know that the files are in a specific directory you can just build the path from the file name and then check if its there. That way you're only querying the file names you download, not every file in the directory. Check out the os.path module. It has an `exists` function which should help you with this specific thing, but it will also make any manipulation involving a file path easier and more robust. In general it is almost always better to encapulate file paths in an actual Path object instead of just using strings, especially in cases like this where you don't have a lot of control over the input. You might even be able to use it to get the file name part out of the URL. Lastly, you may want to URL-decode your file names before you create the files. I think there's something in the standard library that can do this but I don't remember exactly which module its in.

>>21555 >>21557 Oh, also you should never do while i < len(r_json) in python. Even if you need the index its usually better to use the enumerate() method. In this case the index is unnecessary, so just do for e in r_json which gives you the actual list element (the value of r_json[i])

>>21558 BTW the reason for this is because you can iterate over things that don't necessarily have a length, and even if they do using their actual iterator with the 'in' keyword is going to be at worst just as fast, if not faster, than accessing them with their index (for something like a linked list it will be significantly slower because you have to traverse the whole thing from the beginning every time you access via indexing). If you do actually want to get an index array, which does come up occassionally, the "correct" way to do that is "for i in range(len(myarray))"

>>21557 >>21558 >>21559 Thanks a lot lain, that will help me

>>21555 >open(filename, 'wb').write(test.content) https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files https://docs.python.org/3/reference/compound_stmts.html#with with open(filename, 'wb') as f: f.write(test.content) >string_b = string_a.rsplit('/', 1)[1] >filename = curr_url.rsplit('/', 1)[1] https://docs.python.org/3/library/stdtypes.html#str.rpartition >if curr_url.find('/'): https://docs.python.org/3/library/stdtypes.html#str.find """Note The find() method should be used only if you need to know the position of sub. To check if sub is a substring or not, use the in operator""" https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts if __name__ == "__main__": # read SICP >while x < len(r_json): You remove the "file_url" of x number of r_json entries that can live at arbitrary positions, but then you skip the first x entries. for item in r_json: # ... if string_b not in file_list: # body of if curr_url != None: >file_list = glob.glob("*.*", recursive=True) >filename = curr_url.rsplit('/', 1)[1] You only use file names without paths so recursive=True is pointless. >file_list = glob.glob("*.*", recursive=True) >if string_b in file_list: You only use file_list for membership testing so make it a set. file_list = set(...) https://docs.python.org/3/library/functions.html#func-set

what libraries should I consider using if I wanted to make a bot for my private tracker that automatically seeds and leeches torrents based on torrent statistics?

>>21584 requests to talk to the server BeautifulSoup if you have to scrape the data numpy and pandas if you want to do some non-trivial analysis tinydb if you want to use previous snapshots in your analysis You'll probably also want the libtorrent bindings for Python.

>>21584 >>21591 If you use tTorrent or transmission you can control them over an RPC socket. Would probably be easier than using libtorrent directly.

While this is not exactly an on-topic post, I'm going to post it here because said software was written in python (seriously, I don't know where else to post this). The youtube-dl repo got disabled on github and might be taken down eventually. This isn't the first time youtube-dl got into such problem so I don't suppose github is going to take the repo down or keep it disabled forever. I'm not telling you what to do but it is advisable that you clone one of the mirrors and keep a copy somewhere. >repo https://github.com/ytdl-org/youtube-dl/ >DMCA https://github.com/github/dmca/blob/master/2020/10/2020-10-23-RIAA.md There are many mirrors hosted on gitlab (doesn't work with JS disabled) and other git servers as well. https://gitlab.com/explore?name=youtube-dl https://codeberg.org/polarisfm/youtube-dl https://git.disroot.org/41373770180505887198640312520077/yt-dl-clone

>>21595 Github will keep it down until they receive a counter-notice. Then the RIAA will have 10-14 business days to initiate legal action. If they do not, it goes back up. That's how the DMCA is supposed to work but it's worth mentioning that the RIAA has a secret agreement with Youtube where Youtube ignores counter-notices for music claims. It's speculated that the RIAA has indemnified Google to some extent should they have to go to court over it.

>>21595 >The clear purpose of this source code is to (i) circumvent the technological protection measures used by authorized streaming services such as YouTube I can't tell if these RIAA glitterboys actually don't know that streaming and downloading are the same thing, or if they're just pretending they don't know and courts are too retarded to say otherwise.

>>21603 Yes, but downloading in such a way as to circumvent Youtube's alleged protection features and maybe their ToS. In the notice, they say the ytdl docs had examples for downloading a local copy of Taylor Swift videos, among other things. That's the kind of soykaf that sunk Grokster in court if I remember correctly. It may be time for the maintainers to wash their hands of this and for someone else to fork the project.

>>21603 That's an interesting point. There is a clear intent to make it difficult for viewers to save a downloaded copy of videos on the site. That is obvious. However there is also the intent for you to download the video after viewing any given ads. I think they are saying that there are protection measures SECURITY THROUGH OBSCURITY SHIGGY MCFU­CKING DIGGY GOOGLE to stop you from viewing a video until they want to let you. Download or mirror everything you want to come back to, by the way.

>>21606 >downloading in such a way as to circumvent Youtube's alleged protection features and maybe their ToS Wouldn't this be between youtube-dl and youtube? The RIAA doesn't give a soykaf about youtube's ToS, they only care about copyright. Youtube-dl does violate youtube's ToS though, because you aren't supposed to bypass the web UI. Who knows how this would play out in court though. What if I wrote a browser that (a) refuses to load content from google's ad domains, and (b) caches all videos as mp4 files? Just because my browser doesn't work the same way as chromium doesn't make it any less of a browser, so would that be a violation of the ToS? I can't see how it could be, but then how do you draw a distinction between this hypothetical browser and firefox with some addons installed? >examples for downloading a local copy of Taylor Swift videos, among other things I think this is the meat of it. It makes it explicitly a piracy tool. >>21607 >I think they are saying that there are protection measures to stop you from viewing a video until they want to let you. I legitimately don't know what they could possibly be referring to. Youtube has DRM, but not on regular public videos. Do they mean that you have to re-assemble the cached video chunks into a playable file? I wonder if it would still be a copyright violation to copy those chunks out of your cache and play them back with VLC. Also, the way your browser caches files is totally dependent on its particular implementation.

This madness reminds me of this: https://www.cs.cmu.edu/~dst/DeCSS/Gallery/ T-shirts with the source code of youtube-dl's extractors (including the YT extractor itself) coming Soon™.

see also: dedicated GitHub thread >>34949 >>21618 illegal prime when

>>21615 I think ToS is completely irrelevant in this DMCA case, but you raise an interesting point of whether RIAA is allowed to raise the issue. It's not their service's security protections that youtube-dl aimed to circumvent, even if it primarily affects them.

>>21625 oops, meant >>>/Ω/34949

>>21615 The DMCA complaint explicitly mentions a 'rolling cipher' used on VEVO videos (not standard ones) and that there were comments in the source code saying explicitly how to circumvent that. Read that section of the complaint, I'm just typing what I remember.

I invite you all to the dedicated thread as mentioned by >>21628. I was hesitant when I posted >>21595 but now that such thread exists, it would be better if we centralized all the discussion there and didn't derail this one.

>>21591 >>21592 thanks!

>>21595 >I'm not telling you what to do but it is advisable that you clone one of the mirrors and keep a copy somewhere. is it even worth it? youtube-dl versions become deprecated very quickly due to changes in youtube's website. I imagine it wouldn't be long before the whole thing is useless.

>>21640 On the other hand it isn't difficult software to work on. Just somewhat time consming. I'm sure even if the original authors drop it someone else will maintain a fork.

>>21644 If you don't want to work on the YouTube extractor, you can at least use it for other things. Youtube-dl can extract videos from much more than just YouTube.

>>21647 This is true. My point was just that the codebase itself is large but fairly simple. It's just a web scraper at the end of the day. Anyone posting in this thread could probably fix it in a couple hours if a youtube update broke it (just change the name of some api endpoint or css class or whatever). So I'm sure it will find maintainers even if it is abandoned by the authors.

pls help a python newb i am trying to run some python script i downloaded and it fails File "/usr/local/lib/python3.7/dist-packages/collectd.py", line 8, in <module> from Queue import Queue, Empty ModuleNotFoundError: No module named 'Queue' how do i get the Queue module? cant find it in the repos or in pip

>>22161 me again i found that i have this file /usr/lib/python3.7/queue.py is this the one i need but somehow python does not recognize it?

>>22162 ‘Import queue’ Case matters.

>>22161 "Some python script [you] downloaded" is written for python 2, and you are attempting to run it with python 3. Either obtain the python 3 version of the script or run the current script through the python 2 interpreter.

>>16053 Hi, Are there any comprehensive python books with a lot of exercises and projects to work on? When I went to Uni, we used Y. Daniel Liang's books and I enjoyed them. Are there any Python books like this?

>>16952 >weary I'm weary of you not knowing that "weary" means "tired". You should be wary, you might even be leery of fuarrrking this up again.

>>16054 Is it just me or does this seem like it would be better accomplished in golang?

>>23337 This isn't the thread for you to push your pet language.

>>23338 I say it only because concurrency is the entire focus of go.

>>20853 On a tangential note, you should read PicRel (it's a Philosophy book)

i feel like such a nigger retard, I'm learning python 3 and am trying to implement reply threads, does anyone know how to get specific table ids in HTML? ``` class Reply(db.Model): id = db.Column(db.Integer, primary_key=True) title = db.Column(db.String(100), nullable=False) date_posted = db.Column(db.DateTime, nullable=False, default=datetime.utcnow) content = db.Column(db.Text, nullable=False) user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False) post_id = db.Column(db.Integer, db.ForeignKey('post.id')) reply_to = db.Column(db.Integer, Reply_id=id) ``` then im trying to display the reply_to content but i don't know how ``` {% if reply.reply_to %} <p class="article-content" >{{ reply.reply_to }}</p> <!-- something like reply_to == reply_id >reply.content< {% endif %} <h5 class="article-title">{{ reply.title }}</h5> <p class="article-content">{{ reply.content }}</p> ``` help i am Indian

Is Django still being used in webdev or it's as dead as ruby on rails? >>23625 off-topic posts does that, not racism. >>23369 Is this book somewhat related to programming, mate? Or just an interesting philo book?

i feel like such a white supremacist retard who compensates with a firearms fetish for minuscule genitalia, I'm learning python 3 and am trying to implement a list of primes, does anyone know how to get the primes right in the REPL? ``` >>> def stream(): ... data = itertools.count(2) ... while True: ... val = next(data) ... yield val ... data = filter( ... lambda x: x%val != 0, ... data)#divide by primes already found ... ``` then im trying to display the primes but i don't know how to fix the filter ``` >>> list (itertools.islice (stream (), 20)) [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] ``` help i am American

>>23646 Dear cracker spic nigger chink white supremacist jewish subhuman person of color, My guess is that your problem is due to the lambda somehow capturing the val variable by reference; with each iteration, the previously captured val may be getting replaced with the newly assigned val = next(data) in all filters. The following corrected code, where I instead pass val as argument to a surrounding lambda, seems to work: python def stream(): data = itertools.count(2) while True: val = next(data) yield val data = filter( (lambda v: lambda x: x%v != 0)(val), data)#divide by primes already found I hope this is helpful to you, my untermensch friend :)

>>23653 You slightly messed up the indentation there.

>>23654 Oh no! Now no one will be able to understand the explanation given in that post! Surely, the lack of indentation in this copy-pasted snippet of code with one line changed makes it all undecipherable for the feeble minds of those perusing this board. That's very sad and everyone cares deeply.

>>23653 Thank you for helping out a white supremacist retard who compensates with a firearms fetish for minuscule genitalia. I slightly modified the indentation of your code: >>> def stream (): ... data = itertools.count(2) ... while True: ... val = next(data) ... yield val ... data = filter( ... (lambda v: lambda x: x%v != 0)(val), ... data)#divide by primes already found ... then ran it thus: >>> list (itertools. islice(stream (), 20)) [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71] I'm also grateful that you went out of your way to include a list of appellatives that demonstrate the standard of basic human decency that the mods uphold as part of "trying to ensure the quality of lainchan", as stated on the rules page. And don't worry, my friend, I don't think you'll face any consequences at all for breaking rule 3 by using personal attacks against me. However, since like all white supremacists I'm a retard, I'm not familiar with your lambda in a lambda trick, so could you link me to the exact part of the python documentation that elucidates the "lambda somehow capturing the val variable by reference" part of your explanation? And is the "untermensch" term you used a programming term? If it's a programming term, could you explain it in more detail?

>>23656 I think you just need to look at the scoping rules for python. The value of v is set to val when the outer function is applied, and v can only be modified within that function. The fact that they are lambdas is not really relevant. def stream (): data = itertools.count(2) while True: val = next(data) yield val def OUTER(v): def INNER(x): return x%v != 0 return INNER data = filter( OUTER(val), data)#divide by primes already found works just as well. IMO, the clearest solution is to just add val=val: def stream (): data = itertools.count(2) while True: val = next(data) yield val data = filter( lambda x,val=val: x%val != 0, data)#divide by primes already found

>>23655 It seams you don't know too much about python so you probably don't know this, but indentation is very important in this language, being used for blocks and the like, as opposed to keywords or braces.

>>23628 I'm using it at work. As far as I'm aware, it's "old-fashioned" but still in widespread use. I wouldn't expect to see it go away any time soon.

>>23676 Did you actually try to run the unindented code or something? Did you think the code was meant to be run? Did you think other people would run it, and therefore you had to make a comment to warn others about the horrible dangers ahead? Let me propose something: maybe, perhaps, just maybe, the (mostly copy-pasted) code was left in that state because it was assumed that whoever read it would be smart enough to extract the relevant information regardless. I now realize that this was a bad assumption; your post here suggests that not everyone is able to do that.

>>23701 pythondefstream():data=itertools.count(2)whileTrue:val=next(data)yieldvaldata=filter((lambdav:lambdax:x%v!=0)(val),data) Perfectly readable, right? If you weren't an idiot (unlike myself, who is a 300 iq super-genius) you'd be able to piece together where the whitespace goes yourself. It's not my fault I fuarrrked up formatting my code.

what is considered the orthodox python library for handling GET and POST requests and their parameters for a normal web API?

so i wanted to control the xdg-desktop-portal through the dbus api. i chose python because i deemed it to be the easiest for the job. i never really worked with dbus before, so i looked up the references here https://dbus.freedesktop.org/doc/dbus-python/tutorial.html https://flatpak.github.io/xdg-desktop-portal/portal-docs.html so i wrote this small script as a test #!/usr/bin/env python3 import dbus bus = dbus.SessionBus() proxy = bus.get_object('org.freedesktop.portal.Desktop', '/org/freedesktop/portal/desktop') proxy.Close(dbus_interface='org.freedesktop.portal.Request') and it outputs Traceback (most recent call last): File "./nig.py", line 5, in <module> proxy.Close(dbus_interface='org.freedesktop.portal.Request') File "/usr/lib/python3.8/site-packages/dbus/proxies.py", line 72, in __call__ return self._proxy_method(*args, **keywords) File "/usr/lib/python3.8/site-packages/dbus/proxies.py", line 141, in __call__ return self._connection.call_blocking(self._named_service, File "/usr/lib/python3.8/site-packages/dbus/connection.py", line 652, in call_blocking reply_message = self.send_message_with_reply_and_block( dbus.exceptions.DBusException: org.freedesktop.DBus.Error.UnknownMethod: No such interface “org.freedesktop.portal.Request” on object at path /org/freedesktop/portal/desktop i don't know if i'm stupid or what? i mean the reference lists Request as a valid interface, and if i try to assign that interface to a variable it goes through, but when i try calling the method it says there's no interface? i don't see anything about there being a different object path for this in the reference either. what is happening? am i having a wrong assumption, or made a mistake somewhere?

>>24538 urlib if you want standard library, requests if you can afford dependencies

>>24547 nevermind, solved it

I've been trying to make a party-style game using python sockets but I haven't worked with [python] sockets in a while so I'm still basically re-learning it

I have a Python question. Maybe you all can help. Let's say I have 3 files. ``` main.py client.py server.py ``` `client.py` and `server.py` both import sys and main imports `clinet.py` and `server.py`. My question is two things. Am I taking a performance penalty importing sys twice? Is there a better way? I used to have all these classes in one file and import everything one time, but the code became hard to manage.

>>24930 >Am I taking a performance penalty importing sys twice? No. https://docs.python.org/3/reference/import.html#the-module-cache """The first place checked during import search is sys.modules. This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths. So if foo.bar.baz was previously imported, sys.modules will contain entries for foo, foo.bar, and foo.bar.baz. Each key will have as its value the corresponding module object. During import, the module name is looked up in sys.modules and if present, the associated value is the module satisfying the import, and the process completes. """

I just found a bug in my current Python project. It was caused by my own misunderstanding of function definition semantics. Indeed, the docs do warn (bold theirs): Default parameter values are evaluated from left to right when the function definition is executed. But I missed that, and assumed default parameters were evaluated anew for each function call. So, I had a few functions that computed some time-dependant value (for example, the time since a file was last modified), and for brevity I made it so the reference timestamp argument could be omitted and default to the current timestamp. My signature would look like def file_age(path, ref=time.time()): and of course the result was that the function would always compare against the time it was when the program started up. Figured I'd share. Watch out also if you use mutable objects for your default values, it is suggested instead to use a default value of None and initialise your object inside the function's body. I guess it's the things you don't know you're assuming that get you. https://docs.python.org/3/reference/compound_stmts.html#function-definitions

>>18467 this is really good, lainon. thank you for this

>>25331 I'd say this is less of assuming is what gets you. But more of you had the correct idea about how optional arguments ought to work. However for your displeasure python acts in very poor ways often, and so it betrayed ones expectations. In general I expect python to expect very poorly and thus not get taken off guard. In better languages this happens less often as it's less poorly designed. This type of behavior has many negaitve implications for how recursive functions or helpers act.

does anyone ever complain that python has Tk/Tcl in its standard library, including a fully functional tclsh8.6 binary? and it's sort of ironic; python was largely responsible for killing the popularity of Tcl (and Perl), but now thanks to python Tcl is basically omnipresent even if <0.01% of python devs even know what the hell it is import tkinter tclsh = tkinter.Tcl() scr = ''.join(open("script.tcl", 'r').readlines()) tclsh.eval(src) tclsh.eval("puts lololololololloleverythingisastring") Just asking, since I *really* like this (I prefer scripting in Tcl whenever possible) and hope there aren't any plans to drop Tcl and do a pure python Tk interface. For once, I'm actually hoping that bloat wins...

>>26090 tclsh.eval("puts lolololololollol you didn't close script.tcl") https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files https://docs.python.org/3/reference/compound_stmts.html#with

>>26091 yeah yeah i was just copying my ipython history for a basic demo. I have a lingering aversion to using `with` interactively, cuz I didn't know you could do a oneliner until recently like this: with open('showboard.tcl') as f: src = ''.join((f.readlines())) still looks like a syntax mistake to my eyes

>>26091 also since now it's my turn to nitpick u, your tcl code gives a syntax error: puts lolololololollol you didn't close script.tcl should be puts lolololololollol\ you\ didn't\ close\ script.tcl assuming if you're just trying to show off how Tcl doesn't need quotation marks around strings

opinions on venv vs. virtualenv? I'd prefer to just use venv since it's in the standard library, and it seems like it's good enough for Guido's needs, since he says it's all he ever uses.

>>26106 I've always just used venv, it has all I could need and is easily scalable

In python, I often like having a function return a list by using yield, instead of having to append to a list object and returning that. However, because yield returns an iterator instead of a list directly, I have to make a second function that converts the output to a list object explicitly. Any way to avoid needing to do this? def parse(lines, some_parameter): return list(_parse(lines, some_parameter)) def _parse(lines, some_parameter): #... for line in lines: #... yield foo(line, some_parameter) I thought about defining the _parse inside of the previous function, but that means that the python interpreter has to construct a new function every time the outer function is called (AFAIK).

>>18467 looking good lainanon! ill try read it this week

>>18467 >I'll take the risk. I hope this helps somebody. Based.

>>26138 def listify (f): @ functools.wraps (f) def ret (*args): return list (f (*args)) return ret @ listify def parse (lines, some_parameter): ... https://docs.python.org/3/reference/compound_stmts.html#function-definitions https://docs.python.org/3/library/functools.html#functools.wraps

>>26142 Where do you typically define helper functions like this? I've usually put things like this in a "util.py" or "misc.py" module, but I've heard that doing that is poor practice.

>>26171 If it needs to be used by several files I try to follow the standard library so listify would go into morefunctools.py.

>>26176 I see. You treat your helper functions as being a hypothetical extension to the official/standard functools module. That way you can just piggyback on their conventions, even though your utility module is actually domain-specific to the application you wrote it for.

Are there any good criticisms of python other than speed and whitespace?

>>26188 It's not a well designed language. It has many things (anonymous functions, classes) but implements only the bear minimum of each. It is the jack of all trades, and the master of none. By not excelling at any given area, it is less suitable than any other language given a specific task. Also, the community and the obsession with "pythonic" style is soykaf.

>>26188 I wouldn't consider whitespace a good criticism. I can't understand why anyone would care unless they have to parse it. Good criticisms would include: * Performance in any metric is pretty poor. This goes beyond speed to include things like startup time and memory usage. Major optimizations have been available for years but CPython, the reference and de facto python implementation, will not implement them. * CPython supports threads but not concurrent thread execution. People have been talking about changing this for ten years. * The language has no clear direction. I don't consider its mixing and matching of various features to be a defect at all, but there's little coherence. * Python 2 to 3 was an ugly experience so the community is afraid to do a hard break with Python 3. Some people say thank God, I say grow some balls and get better at breaking changes. * Virtual environments and distribution could be a lot better.

>>26188 >Are there any good criticisms of python other than speed and whitespace? I'd argue that dynamic typing is a big drawback for non-trivial programs, and Python is otherwise fine for non-trivial problems. Static typing eliminates an entire class of runtime errors.

>>26198 >I wouldn't consider whitespace a good criticism. Well, I think it is. Not specifically the fact that whitespace is significant, but the fact that only whitespace is significant, that is, the fact that there are no explicit block end markers. In any language code can get badly mangled from an editing accident, but not all languages are equally vulnerable. If your language has end markers and you paste something the wrong way, there will probably be mismatched markers somewhere and you'll get a syntax error; in Python or Nim, you may get a program that parses fine but contains a subtle bug. If you mangle indentation, with explicit markers the correct one can be easily inferred; with indentation-dependant syntax, it can't (yet it must). Whether this will ever happen to any given programmer depends on many factors, starting with the tools being used (but here, the ease to develop good tools is a factor of the ease to parse the language, so the point about parsing ease is not so irrelevant). Surely lots of you have never experienced these problems, and others have learned to properly configure your tools to avoid them; my point is simply that indentation-dependant syntax makes certain kinds of bugs and annoyances more likely to happen than explicit-marker-syntax, and therefore constitutes a valid criticism. >>26199 I also agree with this point on typing, but lately I'm become obsessed with static typing to the point that I can hardly think in terms of dynamic typing anymore. It is clear to me that I am not subjective on this topic and won't be until I've explored static typing at length and gained a broader perspective on it, so I won't press this point for now.

Uhh Can I grab + or - from a string and use them as the operation?

Good intermediate/advanced resource on Python? I'm looking for things like how the language works under the hood (garbage collection, objects, types etc.) and for things like list comprehension and best practices. Should I just read the documentation maybe?

>>26203 If you have a valid expression in a string you can use eval(string). No idea about using standalone operator inside a string though.

>>26206 Just what I needed. Thanks luv

>16053 >Beginner Questions are also welcome. all youll ever get with python is beginner questions, its a soykaf language for novice programmers to larp that they know programming. I would never use this piece of soykaf for anything more than 200 lines of code, and only for glue scripts. if you are using this day to day, learn a real language and dump this gargabe >dynamic typing, so that you run into thousands of runtime errors >soykafty non-JIT compilier so its slow as ass >has 0 presense on the web or on mobile, no portability at all >no multi line lambdas so cant properly make functions first class without writing them in an awkard way (have to def and then destroy the function, ass) >idiotically broke backwards compatibility with python 3 since guido was too stupid to understand how strings worked. my fuarrrking perl scripts form 30 years ago still work on linux, but now many distros are deprecating python 2 so my 2 scripts dont work wtf >async support barely bolted on and works like ass >written by someone who doesnt have a compsci backgrounnd (guido has a math background), and so made the same mistakes that wall did with perl (people who dont have a compsci background shouldnt make languages). literally the lang was designed to teach basic programming, not to actually be used >stdlib is soykaftier than Java stdlib and broken in many cases literally this lang has no redeeming qualities. we should have stuck with perl for scripting, and wrote all the data science libraries in java, python is literally unbearable and you all should feel bad for perpetuating this soykaf lang.

>>26213 > and wrote all the data science libraries in java, You have no idea what you are talking about. Just hide and go away.

>>26213 why so angry? you don't have to use it python has its advanteges and disadvanteges as any other language be more civil if you want to have real discussion, otherwise fuarrrk off

>>26213 >Python bad >java good There are many principled arguments against python, but many of them would land against Java as well. You do not have a good position to argue from. Also I forget all the best languages were made by CS poeple. I forgot that Lisp, ML, OCaml, Haskell, APL, Forth, Small talk, Scheme, etc. Were all made by CS people and not Math, Physics, Logicians, etc. Yes how could I forget that the most innovative languages were made by people who studied specifically CS!

>>26213 But CompSci is a loser's degree. Winners do math.

>>26203 Alternately, Python's built-in operators are replicated as ordinary functions in module operator (e.g. operator.add, operator.sub). Eval is more straightforward, but carries its own problems; depending on your use case, it may make sense to, for example, make a dictionary that maps an operator's string representation to the associated function.

>>26214 >You have no idea what you are talking about. Just hide and go away. as usual, the python morons never try to defend the criticisms that are brought forth about this soykaf lang since they havent used any langs in depth. Maybe try working with a 1+million line codebase and tell you can do this with python. enjoy having to prod runtime to figure out whats in your soykafty untyped dataframe. python idiots need to realize they arent devs, they are code monkeys. >>26215 >python has its advanteges and disadvanteges as any other language there is literally no advantage to using the lang, inherit in the language itself or its bytecode and VM. It's only redeeming qualities are that it has a large stdlib. is available by default on my *Nix platforms, and some datascience packages are available for it. However, these are outside of the actual language semantics and syntax. The PVM is ass backwards and the lang is perpetually stuck using old bash-like, OOP syntax. FFS we had to move from 2 to 3 since guido doesnt understand how string encoding works. This was horribly short sighted. >>26219 >There are many principled arguments against python, but many of them would land against Java as well. the complaints I listed above are 100% inherit in python that Java does not have a problem with. read my actual criticisms and tell me that Java has these problems. Regardless, even if you aren't a fan of Java, Kotlin is available and is a million times better to dev in than python >Also I forget all the best languages were made by CS poeple. I forgot that >Lisp, ML, OCaml, Haskell, APL, Forth, Small talk, Scheme, etc. You have a fair point here, I misspoke a bit. What I meant was that languages should be designed by actual language designers who have an affinity for language design. Guido does not have an affinity for this, nor ever intended soykaf python to be used as a language. The intent was always to teach basic programming, not to actually use this.

>>26223 to be fair, any one participating in language wars on chinese weaving boards probably arent paid devs

>>26223 Imagine getting into argument over a thing you don't like because you don't like it and needed everyone to know it.

>>26223 there are no actual language designers. its a no true scotsman.

>>26223 >there is literally no advantage to using the lang someone might argue that first class functions are advatage for python or that decorators are more elegant in python or that dynamic typing is better for processing diverse data or that it's easier to refactor python code compare to Java Personally I don't mind python, but I can't stand Java I hate how verbose and repetitive java code is and I would say that python is much better at this

>>26188 Python's use of whitespace as delimiters for scope and line termination doesn't cross the line for me. It's not worst, and maybe a better choice, than automatic semi-colon insertion that languages such as javascript and go make use of. Hunting a bug down due to python's indentation doesn't waste as much time because its mechanism is a simple set of rules. On the other hand, having to carefully read a page that describes where each type of statement can and cannot be cut with a newline, and its own exceptions to those rules, no thank you.

>>26237 I'll add that the most elegant and sane (exceptionless) approach to statement/expression separation is found in Lisp's syntax

Link dump for >>26250 /*** Beginners ***/ https://www.w3schools.com/python/ (quick reference-like tutorial) https://docs.python-guide.org/ https://libgen.is/book/index.php?md5=922B856D66574746B69EC69E67C20A17 (Python Crash Course: A Hands-On, Project-Based Introduction to Programming) https://libgen.is/book/index.php?md5=2699081BC2E3908ECE25013109941028 (Automate the Boring Stuff with Python) (not meant to be read in one sitting) https://docs.python.org/3/tutorial/index.html https://realpython.com/python-pep8/ https://wiki.python.org/moin/BeginnersGuide/NonProgrammers https://github.com/tuvtran/project-based-learning#python https://www.programming-idioms.org/cheatsheet/Python https://www.youtube.com/playlist?list=PLi01XoE8jYohWFPpC17Z-wWhPOSuh8Er- (Python Programming Tutorials by Socratica) https://www.youtube.com/playlist?list=PLBZBJbE_rGRWeh5mIBhD-hhDwSEDxogDg (Python Tutorials for Absolute Beginners by CS Dojo) https://www.freecodecamp.org/news/search/?query=python https://www.youtube.com/c/Freecodecamp/search?query=python /*** 2D ***/ Program Arcade Games With Python And Pygame For Beginners http://programarcadegames.com Pygame Tutorial for Beginners - Python Game Development Course (2 hrs) (Space Invader) https://www.youtube.com/watch?v=FfWpgLFMI7w (note you don't have to bother with an IDE like pycharm, I use notepad++ on windows and nano or gedit on linux) Pygame in 90 Minutes - For Beginners (2-player 2D space shooter) https://www.youtube.com/watch?v=jO6qQDNa2UY Python Tic Tac Toe Beginner Tutorial in PyGame https://www.youtube.com/playlist?list=PLjcN1EyupaQmPuM0Maw41ao2Xkg0U6XEw PyGame - Snake https://www.youtube.com/playlist?list=PLjcN1EyupaQmGd7CFKpLRrLaagDVLH_QF PyGame - Pong https://www.youtube.com/playlist?list=PLjcN1EyupaQnB9-Ovkisq0Ss-1u0gOAv4 PyGame - Breakout https://www.youtube.com/playlist?list=PLjcN1EyupaQlbAyHLsuFIp0n6i_p8XWaO PyGame - Space Invaders https://www.youtube.com/playlist?list=PLjcN1EyupaQkAQyBCYKyf1jt1M1PiRJEp PyGame - Flappy Bird https://www.youtube.com/playlist?list=PLjcN1EyupaQkz5Olxzwvo1OzDNaNLGWoJ PyGame - Tile Based Platformer https://www.youtube.com/playlist?list=PLjcN1EyupaQnHM1I9SmiXfbT6aG4ezUvu PyGame - Level Editor https://www.youtube.com/playlist?list=PLjcN1EyupaQn-x_9QuGIwIESxfV9kNUZf Python - Final Fantasy Style RPG Battle https://www.youtube.com/playlist?list=PLjcN1EyupaQnvpv61iriF8Ax9dKra-MhZ PyGame - Scrolling Shooter Game https://www.youtube.com/playlist?list=PLjcN1EyupaQm20hlUE11y9y8EY2aXLpnv Pygame Tutorial Series by DaFluffyPotato https://www.youtube.com/playlist?list=PLX5fBCkxJmm1fPSqgn9gyR3qih8yYLvMj Gamedev tips by DaFluffyPotato https://www.youtube.com/playlist?list=PLX5fBCkxJmm1JOYwi8RbT4xBhb7vbBvhP Pygame RPG Tutorials by ShawCode https://www.youtube.com/playlist?list=PLkkm3wcQHjT7gn81Wn-e78cAyhwBW3FIc /*** 3D ***/ https://pythonprogramming.net/opengl-rotating-cube-example-pyopengl-tutorial/ OpenGL programming with PyOpenGL, Python, and Pygame by sentdex https://www.youtube.com/playlist?list=PLQVvvaa0QuDdfGpqjkEJSeWKGCP31__wD Learn OpenGL in python by AtiByte https://www.youtube.com/playlist?list=PL1P11yPQAo7opIg8r-4BMfh1Z_dCOfI0y Developing Graphics Frameworks with Python and OpenGL https://www.youtube.com/playlist?list=PLxpdybrffYlPqkCyvvLfvwsaB7CB1r0pV PYGLMaze - Python + OpenGL Example Game https://www.youtube.com/watch?v=T94Ht1I5-Kk https://github.com/Blakkis/PyGLMaze/tree/main/pymaze Make 3D Graphic with Python Opengl Blender https://www.youtube.com/watch?v=qxmBZ8V58kY

which is nicer, IYO? the examples below aren't realistic of course def foo(): if bar: return 1 return 2 def foo(): if bar: return 1 else: return 2 def foo(): if bar: return 1 #else return 2 the first and last one only differ in the placement of a comment

>>26399 Unless it's a guard clause at the top of a function, I like to be explicit about my intent and use an else. I don't mind it when other people don't do this though. I would never make a comment that says #else.

>>26409 >I would never make a comment that says #else. but why not? stop hating youth culture

how do you structure your local packages (i.e. packages you wrote for your own projects)? this is kind of how I do mine: foolib/ setup.py requirements.txt (contains relative imports to other local packages using "-e ../otherlib" or "otherlib @ file://localhost/home/user/projects/otherlib") tests/ requirements.txt (first line is "-e ..") venv/ (ignored by git) ... main.py src/ possibly_some_helper_package/ (have never tried this) ... foolib/ __init__.py bar.py hello.py baz/ __init__.py (needed (?) for baz to be recognized as a subpackage) __main__.py (this lets baz be run as a script using "python -m") my setup.py usually has something like this in it: requirements = read('requirements.txt').splitlines() setup(name='foolib', version='0.0.2', description='doing foo and bar', python_requires=">=3.6", package_dir={"": "src"}, packages=find_packages(where="src"), install_requires=requirements, # to fix "invalid command 'bdist_wheel'" # see: https://stackoverflow.com/a/54833684 setup_requires=['wheel'], ) as maybe you can guess, 95% of this is cargo cultism I scrapped from various projects and SO answers. I get how it works as written, but I don't have any idea how to refactor it. for one, I don't like having to have the name of the package repeated in the "foolib/src/foolib/..." path.

>>26399 IMO first one. Quite sure some linters also point towards this style (unless I'm mistaking it for bash linters), so I suppose it might depend on whether you'd trust the judgement of a linter or not. (Example 1 might even be faster too, but I never tested it - I'd assume some internal optimizations have been put in place to make these the same, but idk).

>>26431 I use a somewhat similar structure (even have a similar setup.py config for reading the requirements.txt file), but I don't have an src folder, just a foolib folder that contains every file used by the package. Yours does seem a lot more organized though... As for requirements in the test directory - in my case, I just have them in the setup.py file: python setup( extras_require={"test": ["pytest", "...insert other packages here..."]}, ) > for one, I don't like having to have the name of the package repeated in the "foolib/src/foolib/..." path. Something tells me that's just standard practice, but I'm not sure. If you're not making a multilib project though, you could probably just use the "simplified" layout I just mentioned.

>>26399 There's also: python def foo(): return 1 if bar else 2

>>26455 a good pointer, thanks. I didn't include that form though because I'm imaging a scenario where the function is pretty long and complex >>26432 >>26409 no love for the third one huh? I think it's the best one though I'd hesitate to use it publicaly. the problem with the first two imo: (1) (direct return): if there's many lines between the guard and the other stuff, and the if the "guard" isn't at the very top line, then it's not obvious at a glance that the function could already have returned later on. (2) (redundant if-else): the "else" clause implies that the conditional could influence the control flow, but it can't. so basically you're using language syntax for commenting, which is confusing since one would expect them to be used separately. which leads me to think the third form is the most "rational" though certainly quirky

>>26473 You're arguing a good case for #else. The explicit else is faster to read, because of syntax highlighting and matching indentation, but I wonder if it can be misleading. If your function is long enough that you want to include such cues to improve readability, then a reader who looks deep enough into it to notice that the else is superfluous could reasonably assume it to be a mistake, and proceed to waste some time hunting for bugs in its body. >using language syntax for commenting That's an interesting thought. I saw nothing wrong with it at first, but now I see how it could be misleading. You can, of course, still have the best of both worlds: def foo(): if bar: return 1 else: # for readability return 2

>>26487 > else: # for readability ah, a fourth possibility which I hadn't thought of... thanks for pointing it out. I like it more than my bare comment approach.

Reimplementing Ray Tracing in One Weekend in Python using Numpy. Enjoying it so far, starting to run into performance issues so I think I'll need to streamline as much as possible into the numpy functions.

Need to webscrape / automate web browser activities? Solution = Python + browser + browser driver + framework Python code calls framework. Framework talks to browser driver. Browser driver commands browser to visit pages and take the actions you specified in your code. Popular frameworks: Beautiful Soup Selenium +others Browser drivers: chromedriver firefox driver Browsers: Google chrome Mozilla Firefox Parsing HTML on static sites mainly? Choose Beautiful Soup. Need to automate complex testing on dynamic sites and scale up to many instances of tests in the future? Choose Selenium From experience Selenium is more flexible and will cover any use cases you have. TIP: Ensure the Browser driver installed is compatible with the Browser version in use.

>>26562 nice. used Ruby for doing this stuff before. might try Python next.

What are some good testing/mocking techniques to know? Especially any blog posts or resources on testing large applications would be good to know.

>>26639 >What are some good testing/mocking techniques to know? IOC containers are useful for testing dependencies. instead of having your program import foolib you can give it mock_foolib when it asks for foolib, and it won't know the difference. that way you can write "proper" unit tests that don't rely on any foreign code outside of your control.

https://www.composingprograms.com A Computer Science introduction with Python based on SCIP

Hello. I'm currently learning python and I'm trying to set up a nvim dev environment. I already have pyright and black, is adding flake8 and pylint an overkill ?

>>28529 I use ALE its compatible with a lot of langs and a lot of linters and even autocomplete, you can toggle things on and off in the .conf until you find what works best and means not installing and configuring things manually all the time.

Just another CompSci introduction book that uses Python: The Practice of Computing Using Python by Willian F. Punch

anyone here had performance issues with python? if so how did you make things faster?

>>28604 Post your code and write a more specific post. Otherwise, install Jruby (sic.)

>>28605 i dont have code to post

>>28604 Yes but its either the programmers fault or just a case to use julia instead. You can also wrap a c library

I made a script to scrape the current silver price off https://www.apmex.com/category/20000/silver-bullion and show it in the terminal. This is my first python script in probably half a year so it was fun to get back into this sort of thing. Still perfecting it so I won't release the code yet. Just made it so you can pass an argument as to how often to update the price. I also ordered some wires and an LCD display to hook up to my Raspberry Pi and hopefully I can display the current price on it. I'll make sure to update and say if it works.

>>28615 Why would you complain about efficiency when you don't even have any problems with it?

>>28669 learn to read. there was no complaint.

Does anyone have any advice or onsight into writing python (or anything else) scripts to help archive the messages on a bbs system? I'm not very familiar with the protocols involved but would like to write a script to connect to a bbs (via telnet most likely as there's telnetlib for that) and then interact through to the messages log, where the program would then view and save every message.

>>29194 sentdex for machine learning stuff

>>26562 I've done this kind of thing before to scrape game soundtrack upload sites for mp3s and not have to pay for premium lol. Pretty useful for that but you're fairly restricted to what you can do since most sites have anti-bot measures. iirc google refuses to work with the firefox webdriver because of this.

>>29359 Could you not just use the same variable name (say “it“) on each line?

>>29359 Something like that ? matches = {list(search("%s{:w}" % query, e['data']['lines']['text']) or [None])[0] for e in dataset if e['type'] == 'match'} - {None} # fool's errand print(matches)

>>29359 >>29360 >>29369 can't put it to rest, figured i could use a functional - pipe approach found existing solutions in the form of https://github.com/JulienPalard/Pipe and such, but figured it shouldn't be that difficult to make your own mini implementation, right? some parts of the code in the quoted post, reimagined: import functools import json pipe = lambda *args: functools.reduce(lambda acc, el: el(acc), args) pipe(open('/tmp/grepped.json'), lambda it: it.read(), lambda it: it.replace("}\n{", "},{"), lambda it: it.strip(), lambda it: '[%s]' % it, lambda it: json.loads(it), lambda it: [e for e in it if e['type'] == 'match'], lambda it: map(lambda e: e['data']['lines']['text'], it), lambda d: print(list(d))) not much of an improvement if i'm being honest

>>29371 I find this latter version quite easy to read (largely because of the more uniform layout). The biggest hurdle I find is the fact that 'it' changes suddenly from a string to a list of dictionary-like objects, each one nested several levels deep - maybe you could split your transformations into several pipes depending on the data they work on? Read the file and pass it to the string-massaging pipe, then pass the result to the JSON pipe that knows about trees and searches for specific patterns, and so on. Also, since some of your transformations can blow up if the data isn't in the right format, you can use that fact to decide where to split and how to group them: arrange your sub-pipes such that each has a pre-condition on what its input should look like, and is guaranteed to work as long as that condition is met. Finally, I notice you've deliberately put only one transformation per line, and I think your code overall benefits from that (my impulse would be to chain a few of them together), but I do believe you could group together conceptually similar transformations without a loss in readability. For example, in the first version you first filter None and then filter 'nil' - I do think these could together and stay readable.

How can I make a string that the user can edit at will? Like: python3 def EditString(string): #Show s on the string and allow user to edit it at will return edited_string s = "Let's all love Lain" s2 = EditString(s) print(s2) The idea would be that this edit would allow the user to edit the text like in a text editor, they could maybe just add "!!" to the end, change Lain to "me" or even the text completely to something totally different. The only hacky way I could think of only works to add stuff to the end python3 s = "Let's all love Lain" + input("Let's all love Lain") But that only really let the user add stuff to the end of it and they can't delete anything from the original text which is not what I want.

>>29397 What you want to do is not trivial. Consider that, if the user deletes a character from the back, you need to 'undisplay' that character to provide feedback. You need a 'cursor' of sorts, that the user can move to edit the string at a given place, and you need to update what is displayed to the terminal in response to user actions. You can't do that with ordinary IO. Readline does what you want, but it also does many things you don't need (like auto-completion and input history). There are bindings for it in the stdlib. Fiddling with it for a few minutes, I haven't found the way to output an 'editable' string on demand, but the up arrow key by default retrieves the latest input entry as an editable item, so if I were you I'd start by looking into what the arrow keys are bound to. Another way to do what you want is to control the whole terminal with something like curses, but that is almost certainly overkill for your needs.

Okay, Lain, gonna try to learn Python using the resources in this thread, wish me luck. I will report in here again once I've learned some things, the thread will probably still be up knowing this site's speed.

fuarrrk this language, python has so many fuarrrking footguns and is a pain in the ass to refactor, but god damn is it fun to write and has such a superb ecosystem. every time i try to leave the python world it just brings me back in, the ecosystem is fuarrrking massive and so approachable, nothing even compares to it. its the only language ive ever hated to love so much

>>30859 >superb ecosystem Yeah, right. Web dev tier "programmers" everywhere, soykafty documentation and a package management situation so pathetic it makes npm look good. The lang itself is garbage made by a man who hates functional programming and which was never made to scale beyond scripting (look Ma, we added type annotations so we can be a "serious" PL!).

>>31491 > What's the best resource for learning Python if you already know basics of programming in Bash? I think I followed the learn python the hard way back in the day. Hard to say whats the best but that ones fine. > Any tips for using Python for system maintenance and replacing Bash scripts? I don't know why you want to do this, but whatever. You can run shell commands just the same as bash with the subprocess module. For some things there are special python libraries. > How to use Pip properly? If I want to make a program but I need library from Pip, how do I package my program properly? Do I just tell the user to install it from Pip? How do I automate it? your distro should have a pip package. Just use it. Your distro also will have many python libraries packaged for your distro. Use these in prefference to the same packages avaliable through pip. If you need to install things through your program, pip can be scripted in python. Its also possible to install things for a user specifically rather than system wide, thus evading need for root. personally I usually avoid pip because its generally been a pain to work with for me. often easier to just manually build&install packages by hand or through my very own scripts. - Is there a good interactive environment for Python? Should I learn to use Jupyter? Ive never used jupyter. I guess I don't even know what an interactive environment means. You can test scripts and stuff using the python -i command, which gives you an interactive shell after executing the file argument, so if you write up a file with i.e. a bunch of functions you can access them thus. Also, open a shell in a directory with demofile.py and run the command 'import demofile', and likewise you'll have access to its classes, functions, etc. rest of the questions I aint got no good answers for

>>31491 >What's the best resource for learning Python if you already know basics of programming in Bash? The official docs have a tutorial. I think it's written with you in mind. Start there. Remember that anyone selling introductory material is a filthy grifter. >Any tips for using Python for system maintenance and replacing Bash scripts? Get familiar with argparse and with the various ways of starting and interacting with processes. >How to use Pip properly? If I want to make a program but I need library from Pip, how do I package my program properly? Do I just tell the user to install it from Pip? How do I automate it? Packaging is complicated. For simple things, creating requirements.txt and installing dependencies from that is typical. For more complex things, you want something like poetry. >Is there a good interactive environment for Python? Should I learn to use Jupyter? The normal python REPL, ipython, jupyter, bpython. Python's interactive development support is above average. >Are there some tools to lint/analyze the code? Are there some tools that help debugging? flake8 for linting. pdb for debugging. >Are there some important Python related sites or blogs/youtube channels that I should be aware of? No. Videos are good for demonstration and suboptimal for everything else. Maybe check out pymotw, dive into python, and automate the boring stuff. All popular and widely recommended. >Is there a good site for exercises or easy projects? exercism.io for exercises.

Bump! I love Python !!! Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>

Could any Lainon suggest me a book for learning Python considering that I'm not new to programming? I am looking for a general book about the language itself maybe with some later chapters including numpy, but not really necessary, but not the 600 pages long books that assume it is baby's first language. I can program in C, C++, C#, Common Lisp and Perl, and I'm interested in learning Python because my coworkers are stupid and cannot read nor maintain anything but basic stuff, so I I plan to leverage some work from them if I make their life's easier.

>>32432 Mark Lutz book "Programming Python" is good but huge. I heard that "Fluent Python" is also good.

Bump! Where the pythonistas at? What are you guys working on?

I have been working on a tool to manage my investments. I want to start building visualizations and a GUI to make the tool more user friendly. I would like this UI to be able to display graphs and charts, Have a nice way to display data and be able to be responsive. I am most comfortable with webdev so I am not sure if there is something that supports CSS

>>37586 >I would like this UI to be able to display graphs and charts >I would like this UI to be able to display graphs and charts, Python has only Matplotlib and outrageous GUI libraries (and PyQT which probably doesn't work as intented). Do not even think about it, make GUI as a web page. There's a lot of JS and CSS libraries that draw charts.

>>37620 Agreed, although as kind of a middle-of-the-road approach, you could have the GUI be in JS and call a Python backend that renders the relevant graphics with matplotlib.

>>37586 Maybe PyQt if you're looking for an application with menus and stuff. Plotly is a pretty and interactive library for charts that supports python and javascript: https://plotly.com/python/time-series/ https://plotly.com/javascript/time-series/ There might be libraries that are more interactive or you could try drawing candlestick and line charts yourself in a PySDL2 application or maybe a PyQt application. As for web-like stuff I've written a simple nodejs application although it didn't use a gui. I've never used electron although writing a single threaded application in javascript is tempting especially with all the libraries. Although I'm not sure if the drawing thread can be run in parallel with the application or not. It might take getting used to running the window on the same thread as the application. So if I were reading a file or calling a procedure that takes a long time in javascript I might want to make the procedure return a promise instead of possibly blocking the window drawing thread. https://stackoverflow.com/questions/50264553/does-electrons-main-thread-block-browserwindow It seems like the performance can be improved by having the backend start a websocket server and having the frontend connect to it. https://medium.com/actualbudget/the-horror-of-blocking-electrons-main-process-351bf11a763c Also the electron approach used above might be taking up around 200MB of memory.

>>37639 Also if you're distributing the source code of the software then it might be convenient to include build scripts and libraries for platforms that call an application like cx_Freeze to freeze dynamically linked projects by including things like a python interpreter or maybe even libraries along with the executable. https://cx-freeze.readthedocs.io/en/latest/index.html

>>37586 There's tkinter. https://www.youtube.com/watch?v=ibf5cx221hk

>>37586 i really like wxpython for some reason, its mega-bloat but i like things that have everything i want + it looks native. im writing a similar application though the graphs can be less responsive, i was thinking of using matplotlib with the wxwidgets backend since theres a tutorial for it but is that now a bad idea?

I was reviewing concepts, and before I knew it, I had all this written down. I share it because maybe it can be useful to someone who is just learning the language and also, being honest, I don't want to upload it to GitHub but I don't want it to get lost either. Criticism and advice are very welcome. Without false modesty, I am pretty mediocre. - """ Over-engineered 'Guess the Number' game. You will be playing against a smug computer. """ from os import system from sys import platform from random import randint class Rival(object): def __init__(self, secret_number: int): self.secret_number = secret_number def mock_player(self, player_input) -> str: """Gives hints in case the player has chosen a higher or lower number according to the integer object assigned to [secret_number] by the [randint] method.""" if player_input < self.secret_number: return "\nToo low, lo~ser!" elif player_input > self.secret_number: return "\nToo high, lo~ser" def tell_attempts(self, attempts: int) -> None: """Attempts are associated with the chosen game mode.""" print(f"You still have {attempts} attempts left. Keep trying!\n") def rival_win(self) -> None: print("\nSorry! You have lost, lo~ser, lo~ser!\n") def rival_lose(self) -> None: print("\nGrrrr! I hate you! I don't want to play with you anymore, idiot!\n") def game_mode(attempts: int, option: 'str' = 'e') -> int: """Sets the number of [attempts] according to the selected [option]. If the player chooses an undefined value, 'easy' will be the default mode.""" if option == 'm': attempts = 5 print(f"\nI will try not to be so hard on you! You have {attempts} attempts.") return attempts elif option == 'h': attempts = 3 print(f"\nSo you're brave, eh? You will regret it. You have {attempts} attempts.") return attempts else: attempts = 10 print(f"\nAre you a coward? How amusing... You have {attempts} attempts.") return attempts def detect_os_and_clear() -> None: """Detects the player's OS and executes the appropriate command to clear the terminal display.""" if platform == 'linux' or platform == 'darwin': system('clear') elif platform == 'win32': system('cls') else: print("System unknown") def main() -> None: # Game configuration variables. victory: bool = False player_input: int = 0 attempts: int = 0 # Rival configuration. rival = Rival(randint(1, 100)) # Game starts. detect_os_and_clear() print("\nLet's play a game!") print("You have to guess the number I'm thinking of!") print("I will think of a number between 1 and 100.") print("How hard do you want the challenge to be? 'Easy' is default:") option: str = input("[e]asy | [m]edium | [h]ard >>> ").lower() """The [lower()] method helps to ensure that, in case the player enters a capital 'M' or 'H'as input, the option is not detected as if they had entered easy mode.""" detect_os_and_clear() attempts = game_mode(attempts, option) # Attempts are associated with the game mode. while not victory: try: player_input = int(input("What number I am thinking of? > ")) if attempts == 0: detect_os_and_clear() rival.rival_win() break # The game ends and the player loses. if player_input == rival.secret_number: victory = True # Ends the cycle with the player's victory. if player_input != rival.secret_number: detect_os_and_clear() print(rival.mock_player(player_input)) rival.tell_attempts(attempts) attempts = attempts - 1 except ValueError: # In case the player enters a [str] object instead of an [int]. detect_os_and_clear() print("\nThat is not a number!\n") if victory: detect_os_and_clear() rival.rival_lose() # The game ends and the player wins. if __name__ == '__main__': main() - Feel free to use it if you wish to do so.

>>37620 PyQt is genuinely great, imo. I'd mark it (and/or PySide6) as preferable to use to normal C/C++ Qt5/6. I'd use it if I were doing anything short of drawing 3D models. Add PyQtGraph for nice graphs, use QThread for real multithreading, it's just nice and quick ¯\_(ツ)_/¯

Python is my favorite language. Honestly, I have a lot of fun programming with this tool. Even while solving the most exasperating problems, I have a good time doing it. On top of that, I love that you can prototype quickly and efficiently. I also love the object-oriented paradigm of the language (please don't lynch me). Really, I could write a whole wall of text on why I absolutely love this language... But, like everything in life, it's not perfect; there are a few things I miss from other languages while writing code using Python. First, the classic one: Python definitely needs a keyword to close blocks like 'end' or 'done'. Seriously, in very large code it can be pretty difficult to identify where the indentation ends. According to the PEP 8 style guide: Surround top-level function and class definitions with two blank lines. Method definitions inside a class are surrounded by a single blank line. Extra blank lines may be used (sparingly) to separate groups of related functions. It more or less helps, but the truth is that it is still difficult to differentiate blocks; maybe it is because I am a little nearsighted. Also the documentation, I believe, could be better organized. In other languages the documentation is mostly written in a format similar to this: Function/Method: hello_user(name, age) Description: Initial greeting to an user showing their name an age. Parameters: name -> The user's name. age -> The user's age. Return value: String greeting the user. Can be modified to show host system and current time. But in Python the technical details are written as if it were a book that you read from top to bottom; I don't know if I'm making myself clear: hello_user(name, age) Returns a string greeting an user. It shows their name and age. Also, it can be modified to show the host system and current time. Although I admit that this is purely subjective and a matter of taste, I guess... I was just writing code and wanted to get this out of my head.

>>42163 What's the least annoying way to deal with dependency management and versioning with Python nowadays? I rarely use it because that part annoys me. I'd rather just write C or Go most of the time if i have to rewrite my code every 4 months cause someone fixed a bug. All of them seem abandoned to some degree, or niche things from literally whos.

>>42164 >dependency management Yup, with Virtual Environments: Description from Real Python: Creating a Python Virtual Environment allows you to manage dependencies separately for different projects, preventing conflicts and maintaining cleaner setups. With Python’s venv module, you can create isolated environments that use different versions of libraries or Python itself. venv module is already installed by default, and to create a Virtual Environment you should use the command: python3 -m venv <directory> To activate it: source yourvenv/bin/activate Your prompt should change to something like this: (yourvenv) username@hostname:~/<directory>$ To deactivate it, just do: deactivate By the way, it is important that you create a file called requirements.txt and store all the packages that you use in it, because in this file all the dependencies of your program will be stored. Format is: # This is a comment. # It is possible to specify requirements as plain names. beautifulsoup4 # The syntax supported here is the same as that of requirement specifiers. docopt == 0.6.1 To install the packages: python3 -m pip install -r requirements.txt >versioning First and foremost: Never, ever make the mistake of adding an external repository (in the case of Python, deadsnakes). This applies not only to Python, but to software in general. Seriously, you can mess up your package manager configuration. Once that is clear, handling different versions of Python is not complicated. pyenv is your solution! I could write the whole process myself, but this article explains it beautifully: realpython (dot) com (slash) intro-to-pyenv (sorry for this format, it is to avoid spam) >niche things from literally whos To be honest, this is something that involves all languages, ha, ha. The Python Package Index does a great job filtering out malicious packages, so don't worry about that. I hope I've helped you! Whether you decide to give Python another try or not, I wish you success! For someone more experienced than me: If I'm wrong about something, please correct me; I'm pretty mediocre.

>>16054 1. The else is unnecessary, the raise Exception leaves the function. 2. You should be searching a set, not a list. Set has O(1) search-times, list has O(n). 3. You'll run into issues whenever you attempt to use concurrent.futures to do...anything complicated. Just learn how the recent bits of asyncio work. If you're going to say "but I don't want to color all my functions async" you don't need to, asyncio solves that with runners. Regarding the last question, you can use functools.partial calls to segregate internal functions into separately-defined objects. There are other ways but that one tends to work well enough.

>>42164 >>42165 Venv is the immediate-solution, there are tools like pipenv, uv, poetry, hatch, PDM, and Rye. Start with pipenv (it's pip+venv), then figure out uv (it's a bit more), and bother with the rest as needed.