목록분류 전체보기 (126)

오지's blog

snowflake rag구성시 csv파일 chunker

create or replace function csv_text_chunker(file_url string)returns table (chunk varchar)language pythonruntime_version = '3.9'handler = 'csv_text_chunker'packages = ('snowflake-snowpark-python','pandas', 'langchain')as$$from snowflake.snowpark.types import StringType, StructField, StructTypefrom langchain.text_splitter import RecursiveCharacterTextSplitterfrom snowflake.snowpark.files import Sn..
개발노트/Python 2024. 10. 10. 13:50
dict를 넘길때는 double asterik

connection_args = {"user":SNOWFLAKE_IMED_DB_USER ,"private_key":pkb ,"account":SNOWFLAKE_IMED_DB_HOST ,"warehouse":'GCIMED_WH' ,"database":'GCIMED_DB' ,"schema":'DM'}with snow.connect(**connection_args) as conn:
개발노트/Python 2024. 6. 17. 10:31
DateTimeOffset.UtcNow.ToUnixTimeSeconds() python변환

import datetimefrom pytz import UTCoriginal_time = datetime.datetime(1970, 1, 1, 0, 0, 0, tzinfo=UTC).timestamp()now = datetime.datetime.now().timestamp()
개발노트/Python 2024. 6. 13. 15:13
mstr에서 Error type: Jdbc error. Jdbc operation attempted: Connect. [42601:200045: on SQLHANDLE] Private key provided is invalid or not supported: : java.security.InvalidKeyException: IOException : DER input, Integer tag error.구성 요소 처리 방법에 오류 있음: QueryEngin

JDBC;DRIVER={net.snowflake.client.jdbc.SnowflakeDriver};URL={jdbc:snowflake://xxx.ap-northeast-2.aws.snowflakecomputing.com/?user=xx&password=xxx&warehouse=xxx&db=xx&schema=DM&private_key_file=/app/mstr/MicroStrategy/keys/xx.p8&private_key_pwd=xxx};CLIENT_PREFETCH_THREADS=10;fetchsize=160; 아래로 변경 JDBC;DRIVER={net.snowflake.client.jdbc.SnowflakeDriver};URL={jdbc:snowflake://xxx.ap-northeast-2.aws..
개발노트/서버 관리 2024. 6. 5. 10:33
aws ec2에서 인바운드 ip추가해도 접속안되는 오류 해결

ip를 추가해도 접속이 안되서 알아보니 vpc의 라우팅테이블에도 추가해야 한다.
개발노트/서버 관리 2024. 5. 27. 10:05
airflow에서 sftp로 파일 여러개 다운받으려고 할때

https://sftp_multiple_files_download_operator.py/ import os from airflow.exceptions import AirflowException from airflow.models import BaseOperator # from airflow.contrib.hooks import SSHHook from airflow.providers.ssh.hooks.ssh import SSHHook from typing import Any class SFTPMultipleFilesDownloadOperator(BaseOperator): template_fields = ('local_directory', 'remote_filename_pattern', 'remote_hos..
개발노트/airflow 2024. 4. 19. 14:09
distinct와 order by를 함께 쓰려고 할때

SELECT REG_DATE FROM ( SELECT DISTINCT(REPLACE(SUBSTR(CNSL_REG_TME, 1, 10), '-', '')) AS REG_DATE FROM GCWB_WDB.ODS.O_CS_CNSL_HST ) ORDER BY REG_DATE DESC; 하나씩 풀어간다
개발노트/데이터베이스 2023. 6. 28. 16:22
쿼리 수행시 ValueError: unsupported format character 'I' (0x49) at index 69 에러 해결방안

result = engine.execute(f"SELECT TABLE_NAME FROM information_schema.TABLES WHERE ENGINE LIKE '%InnoDB%' AND TABLE_TYPE='BASE TABLE';").fetchone() ValueError: unsupported format character 'I' (0x49) at index 69 에러 발생 % 대신 %%으로 수정하여 해결 result = engine.execute(f"SELECT TABLE_NAME FROM information_schema.TABLES WHERE ENGINE LIKE '%%InnoDB%%' AND TABLE_TYPE='BASE TABLE';").fetchone()
개발노트/Python 2023. 4. 25. 15:08
AttributeError: 'OptionEngine' object has no attribute 'execute' 에러 해결 방안

df = pd.read_sql_query(sql = sql, con = mysql_conn) AttributeError: 'OptionEngine' object has no attribute 'execute' 다음과 같이 변경 필요 df = pd.read_sql_query(sql = text(sql), con = mysql_conn.connect()) reference. https://levelup.gitconnected.com/how-to-fix-attributeerror-optionengine-object-has-no-attribute-execute-in-pandas-eb635fbb89e4 How to Fix AttributeError: ‘OptionEngine’ object has no attrib..
개발노트/데이터베이스 2023. 4. 19. 16:55
airflow에서 select값을 가져오는 방법

sf_hook = SnowflakeHook(snowflake_conn_id="SNOWFLAKE_CONN_ID") sql = f"SELECT DISTINCT(REPLACE(SUBSTR({date_column_name}, 1, 10),'-', '')) FROM {table_name} WHERE SUBSTR({date_column_name}, 1, 10)='{yesterday.strftime(date_string)}';" conn = sf_hook.get_conn() cursor = conn.cursor() cursor.execute(sql) result = cursor.fetchone() 참고 https://stackoverflow.com/questions/53565834/fetch-results-from-..
개발노트/데이터베이스 2023. 4. 18. 10:52
Prev 1 2 3 4 ··· 13 Next