by + agent

In a passive clause, we usually use by to introduce the agent – the person or thing that does the action, or that causes what happens. (Note, however, that agents are mentioned in only about 20 per cent of passive clauses.)

• All the trouble was caused by your mother.
• These carpets are made by children who work twelve hours a day.

Some past participles can be more like adjectives than verbs (see here): for example shocked, worried, frightened. After these, we often use other prepositions instead of by. (And such structures are not really passives, but examples of be + adjective.)

• I was shocked at/by your attitude.
• We were worried about/by her silence.
• Are you frightened of spiders?

With is used when we talk about an instrument which is used by an agent to do an action (see here).

• He was shot (by the policeman) with a rifle.