1/2 논문 중간 점검

1. 주제

- 일단 알고리즘 구축으로 가고,

- 아동문학 구축 (혹은 교육자료구축) 주제는 그대로 가져가서 중앙사랑 출품 (논문 형식으로 쓰기는 할 건데 심사는 안 받을 거 같음)

 

2-1. Algorithm for Multimodal 진행사항

도식화

 

고민 중인 문제

1) 멀티모달 도달까지의 한계점

1-1) 멀티모달은 이미지 투 이미지 지원, 단일 이미지 모델은 그렇지 않은 경우가 많음

Kling ai 레퍼런스 기능 활용 

or

https://ml4a.github.io/guides/Pix2Pix/

Pix2Pix

픽스 투 픽스 코드 연동

 

1-2) image to text 기능 구현 여부

 

1-3) 멀티모달에 image가 아닌 다른 종류의 input이 들어온 경우의 구현 여부 

ex) 엑셀이나 csv 파일 생성 어떻게...?

 

단순히 LLM + 이미지 모델의 문제가 아님, 

음성도 있을 테고, 영상도 있을테고

멀티모달이 구현 중인 기능을 리스트업해서, 이걸 실행할 수 있는 여러 모델의 API를 연동해야할 것 같음

중요한 건 텍스트 청킹이랑 사용자 입력 classification 을 통한 적절한 API request 을 보내는 작업일 거 같은데

 

1) classification을 위한 classification llm 파인튜닝 (label 값을 각각의 모델로 설정해서 탐지의 느낌으로 가야할 거 같음)

https://modulabs.co.kr/blog/multi-label-classification/

다중 레이블 분류 (Multi-Label Classification)

 

2) API request 형식으로 보내기 위한 LoRA adapter 학습

 

이 필요할 거 같음 -> 둘 다 colab으로 잘하면 할 수 있을 거 같은데 (일단 1번은 가능, 2번도 QLoRA로 하면 가능하지 않을까, 안 되면 1번에 프롬프트 엔지니어링으로 해결하는 방향으로 가기)

 

 

2-2. Child literature creation 진행 상황

아동문학의 독자는 문장을 읽을 수 있는 때부터 성인이 되기까지의 아동
아동문학의 독자는 여타 문학의 독자에 비해 매우 두터운 독자층을 형성

소박, 단순한 단어를 활용해 문장의 길이가 짧고 간결하다, 즉 아동이 이해하기 쉬운 형식과 내용을 가진다
아이들이 잘 알 수 있는 어휘와 아이들의 생활에 관계있고 관심을 가질 수 있는 소재
소박, 단순한 것이 많고, 인간의 자연적인 성품을 강조하는 소박한 정신을 강조
주인공들의 묵상이나 사려보다는 행동을 강조한다
주인공은 대부분 어린아이이거나, 어른이더라도 어린이처럼 묘사된다.
많은 동물이 등장한다
추상적이고 공상적인 요소를 가지고
줄거리에 서술이 치중하면서 디테일한 묘사는 그렇게 많지 않음

그림이 중요한 역할을 한다

사회의 부정적인 면을 보여줄 수 있다

예술성, 흥미성, 교육성, 단순명쾌성, 특수성

 

https://blog.naver.com/xbomb/140148223699

아동문학

https://m.blog.naver.com/mingiai12/222868051430

[유아교육]아동문학의 특징과 기능에 대해

 

 

-----

 

 

1. text 부분

 

일단 파인튜닝이나 adapter 활용이 아니라 프롬프트 엔지니어링으로도 충분히 구현이 가능할 것 같았음

데이터 구축 간단한 거면 지피티로 후루룩 하겠는데 저작권이 걸려버림

text chunking은 가장 글의 흐름을 방해하지 않는 선에서 하기 위해 chapter 단위로 잘라서 LLM 및 이미지 모델에 투입

 

input)

Chapter I IT IS A TRUTH universally acknowledged that a single man in possession of a good fortune must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighborhood, this truth is so well fixed in the minds of the surrounding families, that he is considered as the rightful property of some one or other of their daughters. “My dear Mr. Bennet,” said his lady to him one day, “have you heard that Netherfield Park is let at last?” Mr. Bennet replied that he had not. “But it is,” returned she, “for Mrs. Long has just been here, and she told me all about it.” Mr. Bennet made no answer. “Do you not want to know who has taken it?” cried his wife, impatiently. “You want to tell me, and I have no objection to hearing it.” This was invitation enough. “Why, my dear, you must know, Mrs. Long says that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise-and-four to see the place, and was so much delighted with it that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and some of his servants are to be in the house by the end of next week.” “What is his name?” “Bingley.” “Is he married or single?” “Oh, single, my dear, to be sure! A single man of large fortune—four or five thousand a year. What a fine thing for our girls!” “How so? How can it affect them?” “My dear Mr. Bennet,” replied his wife, “how can you be so tiresome? You must know that I am thinking of his marrying one of them.” “Is that his design in settling here?” “Design? nonsense, how can you talk so! But it is very likely that he may fall in love with one of them, and therefore you must visit him as soon as he comes.” “I see no occasion for that. You and the girls may go, or you may send them by themselves, which perhaps will be still better; for as you are as handsome as any of them, Mr. Bingley may like you the best of the party.” “My dear, you flatter me. I certainly have had my share of beauty, but I do not pretend to be anything extraordinary now. When a woman has five grownup daughters, she ought to give over thinking of her own beauty.” “In such cases a woman has not often much beauty to think of.” “But, my dear, you must indeed go and see Mr. Bingley when he comes into the neighborhood.” “It is more than I engage for, I assure you.” “But consider your daughters. Only think what an establishment it would be for one of them! Sir William and Lady Lucas are determined to go, merely on that account; for in general, you know, they visit no new-comers. Indeed, you must go, for it will be impossible for us to visit him, if you do not.” “You are over-scrupulous, surely. I dare say Mr. Bingley will be very glad to see you; and I will send a few lines by you to assure him of my hearty consent to his marrying whichever he chooses of the girls; though I must throw in a good word for my little Lizzy.” “I desire you will do no such thing. Lizzy is not a bit better than the others; and I am sure she is not half so handsome as Jane, nor half so good-humored as Lydia. But you are always giving her the preference.” “They have none of them much to recommend them,” replied he. “They are all silly and ignorant like other girls; but Lizzy has something more of quickness than her sisters.” “Mr. Bennet, how can you abuse your own children in such a way? You take delight in vexing me. You have no compassion on my poor nerves.” “You mistake me, my dear. I have a high respect for your nerves. They are my old friends. I have heard you mention them with consideration these twenty years at least.” “Ah, you do not know what I suffer!” “But I hope you will get over it, and live to see many young men of four thousand a year come into the neighborhood.” “It will be no use to us if twenty such should come, since you will not visit them.” “Depend upon it, my dear, that when there are twenty I will visit them all.” Mr. Bennet was so odd a mixture of quick tarts, sarcastic humor, reserve, and caprice, that the experience of three-and-twenty years had been insufficient to make his wife understand his character. Her mind was less difficult to develop. She was a woman of mean understanding, little information, and uncertain temper. When she was discontented, she fancied herself nervous. The business of her life was to get her daughters married; its solace was visiting and news. 위의 텍스트가 주어졌을 때, 조건을 준수하여 이를 영어로 아동문학의 어투로 고쳐봐 주어진 조건은 다음과 같아 -너무 길지도 않고 짧지도 않지만, 중요한 내용은 모두 생략 없이 들어있는 길이의 문단이어야 함 -등장인물 간의 대화가 많을 경우 이를 요약 및 불필요한 내용은 생략 가능 -소박, 단순한 단어를 활용해 문장의 길이가 짧고 간결하다, 즉 아동이 이해하기 쉬운 형식과 내용을 가진다 -아이들이 잘 알 수 있는 어휘와 아이들의 생활에 관계있고 관심을 가질 수 있는 소재 -소박, 단순한 것이 많고, 인간의 자연적인 성품을 강조하는 소박한 정신을 강조 -주인공들의 묵상이나 사려보다는 행동을 강조한다 -주인공은 대부분 어린아이이거나, 어른이더라도 어린이처럼 묘사된다. -많은 동물이 등장한다 -추상적이고 공상적인 요소를 가지고 -줄거리에 서술이 치중하면서 디테일한 묘사는 그렇게 많지 않음

 

output)

여기 요청하신 대로 아동문학의 어투로 고쳐본 문단입니다. Chapter I (Revised for Children's Literature) Once upon a time, everyone believed that when a wealthy man moved into a neighborhood, he must be looking for a wife. In a cozy little house, Mrs. Bennet was very excited because she heard that a big house called Netherfield was rented by a young man named Mr. Bingley. “Have you heard the news?” she asked her husband, Mr. Bennet, who was always a bit grumpy. “A rich young man is coming to live nearby! We must meet him!” Mr. Bennet just shrugged. He didn’t see why they had to visit. Mrs. Bennet was determined, saying, “Just think how wonderful it would be if one of our daughters could marry him!” Mr. Bennet teased her. “Why don’t you go? You’re just as pretty as they are! Maybe he’ll like you best!” Mrs. Bennet laughed, “Oh, I’m too old for that now. Besides, it would be perfect for one of our girls!” They argued a little, but Mr. Bennet finally agreed to let his daughters visit Mr. Bingley when he arrived. Mrs. Bennet was so happy. She thought of all the nice things they could do together. Even though Mr. Bennet loved to tease, he cared about his daughters. He knew they wanted to meet the new man and hoped that he would bring some fun to their lives. And so, Mrs. Bennet’s mission began to find a lovely match for one of her girls, while Mr. Bennet just wanted some peace and quiet. 이렇게 바꿔보았습니다. 대화는 간략화하고, 아이들이 이해하기 쉬운 어휘로 간단하게 표현했습니다. 필요한 내용은 모두 포함되어 있으며, 전체적인 이야기가 유지되도록 했습니다.

 

-이와 같이 여러 번 돌려본 후 가장 잘 나온 예시 2-3개 정도를 활용해서 N-shot prompt engineering 을 진행하면, 파인튜닝 없이도 좋은 성과를 낼 수 있을 거 같음

 

 

 

2. image 부분

1. 생성형 ai를 활용하여 다음 프롬프트를 채우도록 함

given_text = """Once upon a time, everyone believed that when a wealthy man moved into a neighborhood, he must be looking for a wife. In a cozy little house, Mrs. Bennet was very excited because she heard that a big house called Netherfield was rented by a young man named Mr. Bingley. “Have you heard the news?” she asked her husband, Mr. Bennet, who was always a bit grumpy. “A rich young man is coming to live nearby! We must meet him!” Mr. Bennet just shrugged. He didn’t see why they had to visit. Mrs. Bennet was determined, saying, “Just think how wonderful it would be if one of our daughters could marry him!” Mr. Bennet teased her. “Why don’t you go? You’re just as pretty as they are! Maybe he’ll like you best!” Mrs. Bennet laughed, “Oh, I’m too old for that now. Besides, it would be perfect for one of our girls!” They argued a little, but Mr. Bennet finally agreed to let his daughters visit Mr. Bingley when he arrived. Mrs. Bennet was so happy. She thought of all the nice things they could do together. Even though Mr. Bennet loved to tease, he cared about his daughters. He knew they wanted to meet the new man and hoped that he would bring some fun to their lives. And so, Mrs. Bennet’s mission began to find a lovely match for one of her girls, while Mr. Bennet just wanted some peace and quiet.""" For the given_text, fill in the following elements that are prompts to be put into image creation ai characters and main components: {} background: {} color and atmosphere: {} additional details: {}

 

output)

characters and main components : Mrs. Bennet, a lively and excited woman in her middle age with curly brown hair and wearing a simple but colorful dress, is talking to her husband, Mr. Bennet, who appears calm and slightly grumpy. Mr. Bennet, an older man with gray hair and a mustache, is seated comfortably with a book in hand, wearing a plain vest and trousers. Mrs. Bennet stands nearby, looking animated and cheerful. background: The scene takes place in a cozy living room inside a quaint, countryside house. There is a small window showing a glimpse of a large house in the distance, representing Netherfield. The room has simple, rustic furniture, including a comfortable armchair where Mr. Bennet is seated, and a small table with a teapot and cups. The walls are adorned with a few modest paintings, adding to the homely atmosphere. color and atmosphere : The colors are warm and inviting, with earthy tones like soft browns, greens, and creams dominating the palette. The room is softly lit by the afternoon sunlight streaming in from the window, creating a relaxed and cozy feel. The overall mood is light-hearted and slightly humorous, capturing the playful exchange between the couple. additional detail : Mrs. Bennet has a look of anticipation and enthusiasm as she talks animatedly, gesturing with her hands. Mr. Bennet, in contrast, has a slight smirk as he teases her, with one eyebrow raised. On the wall behind them, a small clock shows that it is early afternoon. The window view hints at the large house in the distance, Netherfield, adding a touch of mystery and excitement to the scene.

 

2. 이를 image 모델로 전달

계속 테스팅해보면서 최선의 프롬프트를 찾아야 될 것 같긴 함