-
Optuna in multiprocessing of python
Previously, I posted how to parallelize optuna. Back then, I didn’t realize that there’s difficult to debug race conditions such as concurrent database access. My method has changed since then, so I’m posting it again. The basic structure of the approach is writing a loop to get parameters, submit tasks, and collect results in batches.…
Tags:
-
Early stopping and timeout in optuna
I have previously explained how to run optuna in parallel. Here, I’d like to explain how one can apply timeout and early stopping. First thing to understand is the use of Manager. It’s required to safely passing around variables in multiprocessing context. In the code below, success, pruneded, failure counters are passed so that multiprocessing…
Tags:
-
Using list of tuples or list of list in optuna
Optuna, hyper paramerter tuning library, doesn’t allow non primitive (str, int, float, None) type of values in categorical proposal. See Support tuples for Categorical values #4893. The post there has a workaround, and here I’d like to write down a bit more generic approach. The idea is feeding string values of tuples, i.e., [‘(1, 2,…
Tags:
-
Shutting down multi processing in python
When you want to shutdown multiprocessor pool, I find this way the best. I tried cancel_futures, and so on, but they all suffers some weird behaviors due to processes terminated forcefully. Instead, we let all workers finish gracefully First, define a stop latch: When submitting task, pass it: In your busy worker function, check the…
Tags:
-
Recommended config_dict for pydantic.BaseModel
It’s difficult to understand why the below is not the default of the pydantic.BaseModel since they look so basic, but anyhow, these are ones I found and keep using. extra=”forbid” prevents following from passing. protected_namespace=(“protected_”, ) makes it possible to use field name that starts with model_. Without it, model_name field causes a trouble.
Tags:
-
Finding all subclasses dynamically in python
Python provide __subclass__ to find all subclasses but it doesn’t find a class if it’s not imported yet. One solution is dynamically import subclass. Assuming that ‘impl’ subdirectory contains subclasses, this is how one can do that. It’s dump of my current code, but you’ll get the idea.
Tags:
-
화면 캡쳐후 chatgpt에 질문 자동 실행 루틴
갤럭시 루틴입니다. 터치 매크로는 chatgpt 앱을 선택하고 녹화하면 되고 터치 시작 전에 delay를 1200ms 정도로 줍니다. 루틴을 빅스비 단축어에 연결합니다. 결과는 이렇습니다.
Tags:
-
plotly and drawing multiple lines in a single figure
I started using plotly. It renders a nice looking chart by default and interactive. With some efforts, one can return json in the server to render it via js. Another benefit is that charts can be merged. It’s very useful in rendering two different lines, e.g., trainig data + test data in a single chart.…
Tags:
-
Interactive plotly charts on jinja2 + flask
It’s difficult to find explanation on how we can render plotly chart on flask + jinja, so I’m sharing my code snippets. First, in python, given a figure, turn that into json response of flask. The chart data will be served via /get_chart/<chart_name> Prepare js code that can render the given json. Following code inserts…
Tags:
-
twelvedata python api
While I was looking for python api for crypto, I found twelvedata. It’s cheap and seems to be comprehensive. But I don’t like their python api, e.g., using pandas than polars, siliently setting number of rows to fetch even when start date is set. So I stopped using their api and wrote a small code…